OpenSimulator, a new beginning?

[Magnuz Binder]

In the beginning, there was a rather clear and simple vision for what was to become OpenSimulator: Create a virtual world server software that works with the Second Life viewer.

This was a pragmatic vision. Many previous attempts of creating free, open-source, virtual worlds had been halted by the overwhelming need for resources to create both a virtual world server and a viewer. By eliminating the need for developing a viewer, the task was simplified.

Also, the time was right. When the OpenSimulator project got started, in late 2006 and early 2007, Second Life was still hot and functionally sufficient, the open-source, partial clone libsecondlife, later libopenmetaverse, was becoming stable, and Linden Lab open-sourced their viewer. At that time, the man with the vision and a proof-of-concept, Darren Guard (MW), met the enthusiast and libsecondlife work-horse, Stefan Andersson (lbsa71). Soon, in February 2007, two more enthusiasts, Gareth Ellison (gareth) and Adam Frisby (AdamZaius), joined.

Within months, in May 2007, the quartet had something going well enough to catch the interest of IBM, through Sean Dague (sdague), and within a couple months, he had IBM interested enough to dispatch a full team to investigate the possibilities with and help with the development of OpenSimulator. At about the same time as Sean Dague appeared, Charles Krinke (ckrinke) also joined in. The rest is, as they say, history.

So, why bring up this ancient history? Perhaps to give an idea of what is required to make "a new OpenSimulator". This story can teach us a number of things to keep in mind:
1. Developing both a new server and a new viewer can be a road block, because sufficient resources to manage both may not be present.
2. To make things work at all, you need both visionaries, looking beyond the lines of code, and work-horses, willing to work according to directions to realise those visions.
3. To have enough resources in the long run, you will probably need corporate partners. Pure enthusiasm only lasts so far.

Today, neither Second Life (SL) nor OpenSimulator (OS) are as hot and attractive as SL was back in 2006-2007. The competition regarding virtual worlds is a lot tougher. And for most, the special possibilities that SL and OS offers, with in-world creation (SL+OS), running own worlds (OS) and hypergridding (OS), just doesn't balance the very steep learning curve for it. Simpler, socially focused worlds, like IMVU, are far more attractive.

So, can we engage enough resources to create a brand new virtual world platform, both server and viewer? Can we engage enough resources to even create a road-map for OpenSimulator, brand new or a continuation of the old, with better management, communication and collaborations than the present one? These are questions I think need to be discussed and answered before trying to undertake such a project.

[Shy Robbiani]

Thank you Magnuz for starting this thread.

Over the past 6-9 months I've been thinking a lot about both the technical and organizational aspects about the subject. The point you raise is the project organization. You don't put Opensim as the software in question.

Regarding the organization I agree with all the points you mention. The technical details are intended to clarify. I am personally a big fan of GitHub and find that GitHub has everything it takes to manage and document smaller projects. But I also see the weaknesses in relation to larger projects. How far this can be eliminated by organizational measures or software alternatives must be clarified in detail. I'm not worried about this.

You didn't mention it, but of course, it also needs more communication paths, like "office hours", forum communication, G+ groups, mailing lists and still important for most developers, IRC.

I have nothing to add regarding the principles of your thoughts about the organization. It coincides with my own expectations. However, it must not be forgotten that any developer invests its time for its very personal interest and goal in mind. If these interests will be to heavily pruned, there will be no development. It will be a major challenge to optimally define the conditions.

I don't like to say that, but what OpenSimulator concerns as a project and software as a whole, so that's already dead to me!

I would therefore prefer to talk about the Open HyperGrid Project (or whatever fits best) and direct the control away from the dependency of the Linden Lab viewer into a new different direction. Thus I have already spoken: the project affects much more than "just" the simulator, it is about the platform that determines the development of the Hypergrid for the next 10 years and beyond.

This development can not take place as a fork of OpenSimulator. We have 2016. I must not be a prophet to say that the momentum of VR begins in one or two years from now on. It's not yet too late to act but the time to do so has come. I anticipate an exciting time over the next 5 years where I would like to participate in one form or the other.

In summary, I stay largely behind what Lena already said. Only with respect to Opensim users who can not live without Second Life, I do not agree. They still account for the majority of the community that has accompanied my life in the Metaverse over the past 10 years. Don't take me wrong, I'm not concerned about the consumer oriented people not willing to contribute by any mean to Opensim. I'm concerned to bring the more conservative users on board as I'm convinced that their needs need to be considered as well. Most of them have been users in both worlds, Opensim and Secondlife, for many years with huge experiences we can learn and benefit from as well.

That also means for me that Opensim should continue to run on its current status and be saved from the total collapse until we have something new to build on. We need the community for feedback as well.

I'm confident, that with a clear vision and a kind of a red thread leading along defined targets towards the goals, will motivate enough people to actively support the project. On the other hand, Opensim had industrial and University support, is it possible to stem all this without?

I realize that the effort is enormous and at best a simple base can be created in one or two years. But a base no one else has, because no one else has the advantage of that many years of know-how behind them. I'm personally convinced, if this hurdle is overcome then nothing stays in the way for a great outcome. But take a look at HighFidelity to get an impression about what can be done and in order to not to put the expectations too high.

Again, for me, the question is to continue at the level with what we have now or to make a big step into the dark, leaving everything behind us with the big risk to fail and to fall. I'm not interested in preserving the status quo forever. Following Linden Lab is no longer an option. Building on the existing viewer is no option on the long run. As I have nothing to loose, for me the decision is already made!

With a good team and a united vision where working together is fun and not kind of a pain, I will be ready to participate from mid May 2016. I could contribute with approx. 8 - 16 hours per week. Maybe more, it depends, but I don't want to make promises I can't hold.

I am not a leader, too. Sometimes I'm kind of an Evangelist. In general, if there is a vision of which I am fully convinced I can assure long-term support and convincingly represent it.

I see my special skills rather in transversal thinking and the ability to make impossible things possible (in terms of code) than in being a good project manager or software architect. I like to work on particularly tricky problems until they are solved. Is the problem solved I am also happy to return to something else and leave the completion to someone else, who in favor may take the credits. That said doesn't necessarily mean I cannot complete things myself in a timely matter.

I prefer to work in silence, but can also conduct meetings good and efficient (at least as long they are held in German), especially when it's necessary to bring several parties with different views to one common denominator.

My knowledge of programming are in C#, Go (golang) and slightly longer back, Java. Go would, incidentally, for me the preferred language in the field of server and simulator programming. I could also imagine to actively support developers who want to learn Go. From my own experiences I know that it takes about 5 weeks to adopt Go for a C# developer. While the syntax is easy to learn there are some concepts that are difficult to understand for most C# and Java developers. With one exception, that is not relevant here, my experiences in programming are mainly in the field of middleware and server applications.

Of course I can script with LSL. Despite of simple examples, I cannot write programs in C/C++ myself, but I can read and understand them. I'm also experienced in the implementation of API's and interfaces to various systems. Regarding Opensim it has often helped me to take a look into the viewer source code to better understand certain behavior. I don't know why, but for some reason C/C++ doesn't turn me on and one thing I know for sure, it will never become the programming language of my choice.

As long it's not about performance and tuning I have enough knowledge of MySQL, SQLite and PostgreSQL as much as it's needed for OpenSimulator. But don't ask me about complex database design or SQL queries.

And if I am already at the weaknesses, then I can also mention that I have no idea when it comes to graphical user interfaces, design of web pages, etc. Just because I know what HTML, CSS and JavaScript are good for, it doesn't mean that I can handle it.

Oh, almost forgotten... I'm not the one who refuses to write documentation. I'm far away from those who express the concepts without further comments or documentation directly in code. But, I see no need for "over-documentation". I love good readable and explaining comments in the code but I dislike to write separate documentation for people who don't care anyway. Modern systems allow to generate appropriate documentation out of the comments in the code where it, from my point of view as a developer, belongs to. All it needs is to keep on some defined rules and one gets the best documentation almost for free. Similar things are true for the automated testing, another aspect to keep in mind, which is quite well implemented in the OpenSimulator project.

Now, I revealed much more about myself than I wanted. I hope I have provided an idea about how I could contribute. I would love to participate in anything that takes off into the right direction. If it fits, you can count on me.

Shy

[Shy Robbiani]

Wow Magnuz,

I'm surprised about the precise and detailed historical knowledge you have. Thank you for sharing, I've forgotten most about it.

In the beginning, there was a rather clear and simple vision for what was to become OpenSimulator: Create a virtual world server software that works with the Second Life viewer.

This was a pragmatic vision. Many previous attempts of creating free, open-source, virtual worlds had been halted by the overwhelming need for resources to create both a virtual world server and a viewer. By eliminating the need for developing a viewer, the task was simplified.

The viewer defined the protocol. That's one of the most crucial problems we are faced today. We depend on it and the more Linden Lab changes the rules the more we loose functionality as we still highly depend on its kernel.

One option could be to participate with HighFidelity, but for now I'm very sceptical about this.

This story can teach us a number of things to keep in mind:
1. Developing both a new server and a new viewer can be a road block, because sufficient resources to manage both may not be present.
2. To make things work at all, you need both visionaries, looking beyond the lines of code, and work-horses, willing to work according to directions to realise those visions.
3. To have enough resources in the long run, you will probably need corporate partners. Pure enthusiasm only lasts so far.

I agree to all written here, but:
1. Having the freedom to work on both, server and viewer, provides the opportunity to work coordinated side-by-side on both.
2. If the visions from the visionaries are attractive and interesting enough it might even attract new developers.
3. Cooperation with corporate partners and universities is a must and it should be discussed in early stages.

Today, neither Second Life (SL) nor OpenSimulator (OS) are as hot and attractive as SL was back in 2006-2007. The competition regarding virtual worlds is a lot tougher. And for most, the special possibilities that SL and OS offers, with in-world creation (SL+OS), running own worlds (OS) and hypergridding (OS), just doesn't balance the very steep learning curve for it. Simpler, socially focused worlds, like IMVU, are far more attractive.

Why not start with a simpler (less simple than IMVU) socially focused world but set the focus towards a future combined with the possibilities SL and OS offers today at the same time? (think broad - act small)

So, can we engage enough resources to create a brand new virtual world platform, both server and viewer? Can we engage enough resources to even create a road-map for OpenSimulator, brand new or a continuation of the old, with better management, communication and collaborations than the present one? These are questions I think need to be discussed and answered before trying to undertake such a project.

It all falles back to the main question. Can we gain enough influence to make OpenSimulator a better organization than today? And what if that happens, what if OpenSimulator has rebuilt itself? Will the software as is survive?

To me one thing is clear. First it needs a fairly well structured organization with clear goals and rules defined, before any single line of code should be written. Then teams for certain tasks should be defined. And most important, it needs someone to keep all together, to communicate the goals we have in common, someone who represents the organization, a manager able and willing to steer the boat, adopted by the community and the whole team.

[Mareta Dagostino]

Really interesting and impressive thoughts, Magnuz and Shy! thank you for sharing them.

Actually I must say that I have no idea how a new Hypergrid could be defined, which could be attached to simple starter solutions like IMVU and also to the big future app of 2026. But I believe that this decides over hail or fail: If a protocol would be able to port avatars from IMVU to HiFi, from OpenSim to WhatsNext, this could be attractive for more players to implement it into their software. I mean something like HTTP, which runs on different servers and browsers.

HiFi: They refuse any normalized text chat support and focus themselves very much on Headsets and high tech equipment. It would be much work to add the things we need - like chat, groups, IM, grids, maps for example. The thing which is called community. What happens if the company stops development and no one of the remaining free time devs understands their code in the deep? On the other hand they "donate" a viewer and a big amont of the server part.

IMVU: I do not believe that many builders and fashion ladies of OpenSim would use a world like IMVU. Implementation of such a simple world is not too difficult, it may be useful as demonstrator and test bed.

Why not think about existing Open Source frameworks like OGRE to render the client side?

Ciao,
Mareta

[Shy Robbiani]

[...]May be I am wrong but if you think to that technology that make us look like a frogman on the sand it is for me more a question of viewer than simulator. keep in mind that some of the so called players wear glasses and it is not a question of Age[...]

I'm a little bit conservative and don't think I'll ever wear these glasses But, the technology is here and will enable a bunch of new applications that demand for further improvement and new technologies we even almost can't think about, yet.

But, you are right. It's basically the user application, what we currently call the viewer, that plays the most important part. It's the viewer we have to talk about. Whether it's running on my PC on flat 2D monitor or on a smartphone while I'm wearing the glasses doesn't matter, we must think about what we want and how things can work together. We must think and discuss the many use cases to get the whole picture. And finally we must define the single pieces and how they can technologically fit into the puzzle with a long term vision in mind that makes the effort worth.

To provide a picture of what we have I prepared some data. It's not accurate and it doesn't take all aspects into consideration, but it also reflects the importance of the viewer code base. In terms of raw code the typical viewer counts 2 - 3 times more than the simulator.

Linden Lab Viewer: 7338 files, 1'088'455 lines of C++ code (incl. header files) and lines of code in XML files, not counting blank lines and comments.
Firestorm Viewer: 9870 files, 1'524'877 lines of C++ code (incl. header files) and lines of code in XML files, not counting blank lines and comments.
Alchemy Viewer: 7554 files, 1'112'883 lines of C++ code (incl. header files) and lines of code in XML files, not counting blank lines and comments.
Singularity Viewer: 4425 files, 944'213 lines of C++ code (incl. header files) and lines of code in XML files, not counting blank lines and comments.
Opensimulator: 1644 files, 414'956 lines of C# code and lines of code in XML files, not counting blank lines, comments and code borrowed from libOMV.
Opensim/limOMV: 1792 files, 573'850 lines of C# code, incl. code borrowed from libOMV and lines of code in XML files, not counting blank lines and comments.

It should also be taken into consideration that both, the viewer and the simulator, make use of third party libraries. Especially on the viewer side, some of the libraries have come into ages and are difficult to handle.

These numbers are from the sources taken yesterday. They are not necessarily accurate and don't take any other code into consideration (e.g. make files, project files and other code needed for building). They also don't count any graphics files. I cannot guarantee I didn't make a mistake.

Basically I wrote a simple program that traversed the source trees, picked any .cs, .cpp, .h and .xml files, removed any comment and blank lines from C#, C++, header and XML files and counted the files and the remaining lines. The results may vary drastically from the results from other sources, sometimes just because they probably grab in hidden folders, otherwise I can't imagine how they come to their impressive results.

I must admit that I've got a little bit shocked when I saw the numbers. Half a million lines of code Depending on my mood and the preparation I'm able to write between 50 and 200 lines of code per day. If everything is perfectly prepared for coding I make 500 or even up to 1000 line a lines a day without testing, but that really very rarely happened ever in my life.

HiFi: They refuse any normalized text chat support and focus themselves very much on Headsets and high tech equipment. It would be much work to add the things we need - like chat, groups, IM, grids, maps for example. The thing which is called community. What happens if the company stops development and no one of the remaining free time devs understands their code in the deep? On the other hand they "donate" a viewer and a big amont of the server part.

I totally share your thoughts. I have the same fears.

Why not think about existing Open Source frameworks like OGRE to render the client side?

I'm aware of, but I don't know much about OGRE. Of course, instead of reinventing the wheel one should look at what is out there and can be used. On the other hand I think the rendering related part is only a relatively small fraction of the whole project.

I'm even not against a refactoring Opensimulator in a way. But I think when it comes to a redesign, the goals should be defined thinking about what we want to achieve in mind and not how we can make Opensimulator better.

[Christoph Balhaus]

Another approach could be to start with one of the AAA game engines that are now freely available (free as in free beer, not free speech) and add Opensim like functionality to it. The recently released Lumberyard engine, a fork of the CryEngine, would be an example for this. See here http://gamasutra.com/view/news/265425/Amazon_launches_new_free_highquality_game_engine_Lumberyard.php or here https://aws.amazon.com/lumberyard/?nc1=h_ls

They are not open source according to OSI's definition and their license has to be carefully evaluated to not conflict with the goals of such a project, but it would give you top-notch quality, certainly on par with anything that may come from Linden Lab, and additional tools to start with.

While any license restrictions may sound like a bitter pill to swallow, my feeling is, that, whatever the design will look like, some tough compromises have to be made to realistically complete such a project at all and this would have the unique selling point of a potentionally superb graphics.

/Chris

[Shy Robbiani]

Good points.
1) licensing and patents are other points to evaluate and take into consideration. The project needs a lawyer
2) Thanks for the reminder, I never looked at Lumberyard because I thought it's bound to AWS. Amazon does a lot to get game and IoT developers on their infrastructure.

Regarding both, I think it's important to stay independent and avoid to get locked into any important parts of the project we depend on and have no influence on.

Edit 2016/03/04 10:40: Ok, Lumberyard can even be used on own hardware, but not on other AWS like web services. The licensing prohibits to publish Lumberyard source code combined with own development. But it's certainly something to take a look at.

[Christoph Balhaus]

Bummer!

I was aware that the license rules out the use of competing cloud services and not to develop a competing product sounds reasonable too, but not being able to distribute even small changes to the source code seems to makes it difficult to use with any open source project.

/Chris

[Magnuz Binder]

To provide a picture of what we have I prepared some data. It's not accurate and it doesn't take all aspects into consideration, but it also reflects the importance of the viewer code base. In terms of raw code the typical viewer counts 2 - 3 times more than the simulator.

Thank you for pulling out the informative code base data, Shy. It does put into perspective what a task it would be to create a new virtual world echo system with both a new server and a new viewer.

But, things don't have to be black and white. If we start the technical part of the work with defining what we want to achieve in terms of protocols, we can then see what needs to be reworked or replaced in the present Linden Lab based viewers. It may actually not be all that much. I have already started to sketch a little on the flow paths of such protocols based on what I think would be suitable and on discussions and documents in the IETF (Internet Engineering Task Force, http://www.ietf.org/ ) working groups MMOX (MMO/Virtual World Interchange, https://www.ietf.org/proceedings/74/mmox.html , http://www.ietf.org/mail-archive/web/mm ... llist.html ) and VWRAP (Virtual World Region Agent Protocol, https://www.ietf.org/proceedings/77/vwrap.html , http://www.ietf.org/mail-archive/web/vw ... llist.html ).

There is a popularised presentation of VWRAP in IEEE Internet Computing: VWRAP for Virtual Worlds Interoperability ( http://internetmessagingtechnology.org/ ... 010073.pdf )

In a way, this is good timing with my day time work, where we are in the process of rewriting some old production systems about the same size as the OpenSimulator code base.

I made an evaluation of programming languages a while back, and at least for performance-critical parts, the choice was Go. It is 2-3 times faster and more memory efficient than C#, but still about 40% slower than C++. It is very stringent, and forces good programming practice, but also limits creativity some, and can demand big workarounds for some situations. It's rather young to the public, which means finding experienced programmers can be hard, and it also lacks some native libraries found in more mature programming languages.

So far, I only worked through a few tutorials and played around with some examples in Go, but I will need to learn it better soon, and despite the somewhat awkward syntax, I like it so far.

We are rewriting our production systems to use services with RESTful API's and message buses, which I think at least partly would fit the requirements for a distributed virtual world echo system. We are also engaged in writing federation software, which deals with quite a few of the problems a fully distributed virtual world echo system would need to deal with, in terms of distributed authentication, authorization and handling of resources.

So, there are more than one way to approach this problem, but hopefully we can have a discussion resulting in some kind of consensus on how to proceed.

[Christoph Balhaus]

I really would love to see such a project fly and would also like to contribute code to it, within the constraints of my day job. Nevertheless, the more you think about it, the more daunting it looks like. Even though i am sure that, with a clever architecture, you could get away with significantly less code than in the current codebase by reusing existing and mature code from other projects (that applies for the server side, i am not sure about the client).

I have some 20+ years of experience in system programming for embedded systems and Unix, mainly realtime, networked and often close to the hardware (that means C/C++, but i am more than a little excited about Rust these days), for high profile scientific and industrial projects and about 10 years with Java for application programming on Windows and Linux, plus a share of Python, Fortran, shell scripting (Yuk!) and whatever was required to get the job done.

For such a project, to attract the required number of talented programmers to it, I think it would need to be ambitious and find new better solutions instead of just cloning and cleaning up the current codebase. Even if the final product would look quite a bit different from what we are used to today.

Edit: What would be the use cases for such a product? I think it's important to clarify that, before talking about architecture and technical/administrative details. Because, as it stands, Opensim wants to be a framework for everything VR related, which IMHO would seem to be an unnecessary and unreasonably wide scope for a new project. Would it be beneficial to set up an IRC or XMPP channel to bounce off some ideas?

/Chris