Autodeletion of builds & types of buildbot (split from end of maintenance branch) - Page 2

Autodeletion of builds & types of buildbot (split from end of maintenance branch)

Post just about everything that isn't directly related to Spring here!

Moderator: Moderators

ivand
Posts: 273
Joined: 27 Jun 2007, 17:05

Re: The end of the maintenance branch

Post by ivand »

As agreed before, it's based off the maintenance + new GL. New GL is working from Lua (as you saw on the previous picture from me), but the interface is unstable. Old GL works as before.

I was going to put together a PR for the transition branch soon after we start playing on BAR branch at a full scale (BAR switched to BAR branch this week), but now I think I'll do another round of enhancements and API revamping before doing a PR. Tentatively something will be ready by Jan-Feb.
abma
Spring Developer
Posts: 3721
Joined: 01 Jun 2009, 00:08

Re: The end of the maintenance branch

Post by abma »

thanks, i've adjusted the roadmap.

springlobby 0.271 x64 is released. I try to create 105.0 as next step so reuploading maintenance hopefully isn't needed.
User avatar
ThinkSome
Posts: 373
Joined: 14 Jun 2015, 13:36

Re: The end of the maintenance branch

Post by ThinkSome »

gajop wrote: 21 Dec 2020, 04:50 ..GH actions...
I recommend against further increasing dependence on proprietary services. Instead, use something FOSS, like Gitlab or some other stuff, perhaps even a self-hosted slave to a FOSS service. Gitlab also supports auto-sync with GH, so it should work hassle-free.
raaar
Metal Factions Developer
Posts: 945
Joined: 20 Feb 2010, 12:17

Re: The end of the maintenance branch

Post by raaar »

abma wrote: 20 Dec 2020, 18:30
bibim wrote: 20 Dec 2020, 16:26 Yeah, the new static builds will fail on a lot of systems because they require GLIBC 2.29, so I guess it would be a good option to put back build 1553 online as afaik it is the last build which doesn't require GLIBC 2.29.
glibc 2.29 was released 2019-01-31, ubuntu 20.04 has glibc 2.31, debian 10 is to old, debian 11 with an up to date glibc will be released in early 2021. On which distribution / version did you currently test?

the build system isn't touched this often: thats why i want to have a fresh system there. putting back 1553 online should do someone else or i will do it when i find some time.
I'm affected by these issues as well. I've tried to use the 1563 maintenance build with my spads hosts and they won't start:

Code: Select all

20201223004607 - NOTICE   - [SPADS] Initializing SPADS 0.12.11
20201223004608 - CRITICAL - [SPADS] Unable to load PerlUnitSync module (Can't load '/spring/spads/PerlUnitSync.so' for module PerlUnitSync: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /spring/engine/104.0.1-1563-g66cad77_maintenance/libunitsync.so) at /usr/lib/x86_64-linux-gnu/perl/5.22/DynaLoader.pm line 187.
 at /spring/spads/PerlUnitSync.pm line 11.
Compilation failed in require at (eval 25) line 2.
BEGIN failed--compilation aborted at (eval 25) line 2.
any suggestions?

EDIT:
ldd (Ubuntu GLIBC 2.23-0ubuntu9) 2.23 <---------------- i have this glibc version and my OS is "Ubuntu 16.04.1 LTS"


EDIT2: rereading abma's post and facepalming
Super Mario
Posts: 823
Joined: 21 Oct 2008, 02:54

Re: The end of the maintenance branch

Post by Super Mario »

ThinkSome wrote: 21 Dec 2020, 16:17
gajop wrote: 21 Dec 2020, 04:50 ..GH actions...
I recommend against further increasing dependence on proprietary services.
They are not an inherently an evil thing.
Super Mario
Posts: 823
Joined: 21 Oct 2008, 02:54

Re: The end of the maintenance branch

Post by Super Mario »

@abma you planning to use cpp modules soon as soon as gcc 11 is released?
abma
Spring Developer
Posts: 3721
Joined: 01 Jun 2009, 00:08

Re: The end of the maintenance branch

Post by abma »

Super Mario wrote: 23 Dec 2020, 16:51 @abma you planning to use cpp modules soon as soon as gcc 11 is released?

is it on the https://springrts.com/wiki/Roadmap ?
Super Mario
Posts: 823
Joined: 21 Oct 2008, 02:54

Re: The end of the maintenance branch

Post by Super Mario »

abma wrote: 23 Dec 2020, 18:48
Super Mario wrote: 23 Dec 2020, 16:51 @abma you planning to use cpp modules soon as soon as gcc 11 is released?

is it on the https://springrts.com/wiki/Roadmap ?
No, but it IS a huge game breaker, when it comes to compile times. I recall seeing #defines statement in spring rts code when it could be easily replace by templates, which I am certain it hasn't done that already as doing so will increases compile time cost. Last time I compile the engine in 2015 it took me around half hour. I have mess with other languages, one of which is the d language and its module system and I sure do NOT mess the c/c++ header file nonsense thank you very much.

FYI my work place uses C#/java-script as its bread and butter. The C# language has receive significant improvements in terms of performance and language features, over the past years. I was thinking about creating C#/D bindings for spring rts, but it looks like there are significant changes going on behind the scenes and the drama regarding BA that I am not familiar with.

All and all, how's your day doing?
User avatar
AF
AI Developer
Posts: 20686
Joined: 14 Sep 2004, 11:32

Re: The end of the maintenance branch

Post by AF »

Re: GH Actions and proprietary-ness, we can run GH Action runners so we aren't relying purely on their infrastructure, but, the actions themselves run bash/shell commands in an Ubuntu/Debian environment. There's no reason they can't be compiled into a standardised shell script. At that point you can either spin up an Ubuntu 20 VM and run the script, or create a docker file that executes the script.

So the work to move to GH Actions has helped, and will continue to help make things more portable and open. It isn't a proprietary dead end.
User avatar
AF
AI Developer
Posts: 20686
Joined: 14 Sep 2004, 11:32

Re: The end of the maintenance branch

Post by AF »

As an aside, it would be a good idea to take a tagged version and manually create a GH release, and upload the built files. That should secure it from any automated scripts or branch deletion
ivand
Posts: 273
Joined: 27 Jun 2007, 17:05

Re: The end of the maintenance branch

Post by ivand »

There is nothing proprietary in GH actions. It's just a workflow that executes scripts one by one. As AF said any qualified individual can move the script to any platform including self-hosted in one day.

The reasons I chose GH actions were:
  • it's the platform spring and BAR hosted their source code at
  • it had a generous cloud workers compute resource provided at no cost for open source projects (2000 minutes / month)
  • everyone is using it
  • too big to fail, at least for more than several hours.
    Just to compare: the recent buildbot outage lasted two calendar weeks, the springfiles total outage lasted couple of days, the springfiles upload bug exists for a few months already.
I feel like any step towards self-hosted / single-person-managed builder is a huge step back. We will again depend on someone's availability, internet, power, state of mind, etc. Cloud workers provide high level of reliability and most importantly unseen level of transparency. Anyone can see how each step is performed, what the results are and if needed, clone the repo and do the same within several minutes. No on prem solution can provide that.

I hold no role at springrts.com, but my personal fork is not moving anywhere and I advise you to do the same.
User avatar
ThinkSome
Posts: 373
Joined: 14 Jun 2015, 13:36

Re: The end of the maintenance branch

Post by ThinkSome »

AF wrote: 23 Dec 2020, 21:50
ivand wrote: 23 Dec 2020, 22:30
Everything can be ported given enough time. The problem is that this increases dependence, and when dependence is sufficiently increased, the time required to do the porting results in a de-facto vendor lock-in.

I have nothing against cloud workers, they should just be used in an easily portable way (e.g. no modifications other than URL change and a few button presses).


"Everyone is using it" should be a reason in itself to NOT use it. It is a de-facto monopoly.
abma
Spring Developer
Posts: 3721
Joined: 01 Jun 2009, 00:08

Re: The end of the maintenance branch

Post by abma »

when all active devs except me dislike to use the buildbot service: why should i keep it running and update it at all?

what prevents you from creating the 105.0 release with github actions?

this again reflects why nothing happens: to many nonsense discussions which prevents people from doing stuff. i guess its time again to split offtopic stuff from this thread like github-Actions.
ivand
Posts: 273
Joined: 27 Jun 2007, 17:05

Re: The end of the maintenance branch

Post by ivand »

abma wrote: 24 Dec 2020, 06:25 what prevents you from creating the 105.0 release with github actions?
I can't.
Firstly enabling GH actions required quite a bit of changes to various CMakeLists.txt and minor changes to source files.
Secondly I reworked CMakeLists such that spring is built with aggressive optimization flags (like O3) and to enable passing custom CPU mtune/march/SSE flags. This represents significant deviation from "classic spring" and while I have seen zero issues so far, I don't feel like the stable is the place for experiments.

I tried to retroactively apply some of the changes above to the maintenance codebase, but I could not get the build going and debugging why was too much effort, because I felt 105 should be built in the existing way.

If I get it right the gajop's proposal is to switch to GH Actions after 105.

P.S. I hold no role at springrts, so you can pretty much count my opinion weight as zero. As far as I'm concerned the official spring can be built with any means. That said I felt I wanted to underline the obvious advantages of GHAs experienced first hand and point to alarming bus factor of the existing solutions.
gajop
Moderator
Posts: 3047
Joined: 05 Aug 2009, 20:42

Re: The end of the maintenance branch

Post by gajop »

ivand wrote: 24 Dec 2020, 08:02 If I get it right the gajop's proposal is to switch to GH Actions after 105.
Yep. And I'll go one step further and say that we should slowly migrate other parts of our infra to GH or similar alternatives (e.g. mantis -> GH issues, self-hosting engine releases -> GH releases).

While I'm in favor of using open source when possible, transferring the hosting responsibility to a extremely popular, free and well renowned service is a better option.
If we were a very big organization it might make some sense to keep going with our own thing, but since we barely have any devs left, and our infra is a poor state, I don't think one can responsibly argue to keep self-hosting, open source or not.
User avatar
AF
AI Developer
Posts: 20686
Joined: 14 Sep 2004, 11:32

Re: The end of the maintenance branch

Post by AF »

"Everyone is using it" should be a reason in itself to NOT use it. It is a de-facto monopoly.
If Abma gets hit by a bus we're fucked. GitHub actions has millions of people who know how it works and use it everyday, as do the competitors and opensource systems. Our buildbot has 1.
Everything can be ported given enough time. The problem is that this increases dependence, and when dependence is sufficiently increased, the time required to do the porting results in a de-facto vendor lock-in.

I have nothing against cloud workers, they should just be used in an easily portable way (e.g. no modifications other than URL change and a few button presses).
Such work was necessary to get that working in the first place, but that level of portability isn't possible as it's too simplistic. You need to tell the CI environment basic things such as what OS you want, versions of things, when to run etc..

If we wanted to be totalitarian, we could spin up a gitlab instance and host the git repos and all the CI and CD ourselves, but then we would need to set it all up ourselves, and we would still need to port everything from buildbot.

The actual work involved in GH Actions was mainly because of the spring engine, not GitHub. This would be true of any CI system wether it was CircleCI, Travis, or GitHub. 99% of the effort was spent figuring out the undocumented parts of buildbot and the out of date compilation instructions. E.g. Buildbots output and the wiki linux compilation output do not match.

As for buildbot, it's serving a useful goal, but, it's a blackbox. What little documentation there is lacks enough context and isn't very useful on its own. Reusable CI scripts would help enormously, be it in local testing, or easy porting to various CI services or software.
User avatar
AF
AI Developer
Posts: 20686
Joined: 14 Sep 2004, 11:32

Re: The end of the maintenance branch

Post by AF »

As an aside, Abma, if you want buildbot that's your prerogative, but if you feel that an alternative can meet feature parity then there are people who can help, the decision is yours in the end
abma
Spring Developer
Posts: 3721
Joined: 01 Jun 2009, 00:08

Re: The end of the maintenance branch

Post by abma »

AF wrote: 26 Dec 2020, 15:22 If Abma gets hit by a bus we're fucked. GitHub actions has millions of people who know how it works and use it everyday, as do the competitors and opensource systems. Our buildbot has 1.
thats not true. turboss + dansan have root access to springrts.com + several people have full github access to the spring organization and full instructions how to setup the buildbot are written / scripted: https://springrts.com/wiki/Buildbot:LXC

105.0 could be released without me: it just seems that nobody wants to do it.

my personal experience with github is, that as service, it regulary will break your stuff: especially when it hurts the most, greetings from murphy. thats why i personally want to avoid cloud services as it usually costs more time in maintenance.

still i see this discussion as offtopic for this thread. (note: i added GH-Actions to the roadmap, but i won't/can't help with setting it up)
User avatar
FLOZi
MC: Legacy & Spring 1944 Developer
Posts: 6238
Joined: 29 Apr 2005, 01:14

Re: The end of the maintenance branch

Post by FLOZi »

It may be off-topic for this thread (though you can consider this thread as 'future of Spring' in which case it is entirely relevant discussion) but it is undoubtedly an important one and long overdue. Single points of failure are problematic, and you - through years of invested time and dedication, this is not a personal criticism - are a major one for Spring infrastructure.

I carry no weight but I'm totally behind gajop and AF.
User avatar
ThinkSome
Posts: 373
Joined: 14 Jun 2015, 13:36

Re: The end of the maintenance branch

Post by ThinkSome »

AF wrote: 26 Dec 2020, 15:22
I recognise that Abma is the weak link and that something must be done, but not by increasing dependency on proprietary services. GitHub is not the only service out there, similar features are offered by GitLab and SourceHut, which are (respectively) semi and fully free software and can be self-hosted if they go bad. There are also multiple gitlab CE hosts out there where the project could move "at a moment's notice". Gitea and gogs perhaps also offer CI functionality. There are a lot of options out there that do not involve depending on proprietary services.
Locked

Return to “Off Topic Discussion”