860246 - Do Android builds in a 64-bits mock environment

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mike Hommey [:glandium] OoO back on Apr 15]

bug 859341 shows that OOM in the 32-bits environment are likely to start coming in the near future. It would be better to preemptively switch to 64-bits environments than to wait for the tree to burn and have to do it in a rush, blocking development in the meantime. As long as a few low-level i686 packages are installed in such a x86-64 environment (libc, libz, libstdc++, libm, libgcc) the current x86 NDK should work (it doesn't depend on too many x86 libraries). In fact, the 64-bits mock environments we have might already have all that is needed.

Comment 1

[Chris AtLee [:catlee]]

I needed to add these packages for b2g emulator builds (which also require a 64-bit build environment):

"glibc-devel.i686", "libstdc++.i686", "zlib-devel.i686", "ncurses-devel.i686", "libX11-devel.i686", "mesa-libGL-devel.i686", "mesa-libGL-devel", "libX11-devel"

I suspect the list would be similar for android builds. Maybe we should switch all builds over to 64-bit?

Comment 2

[Mike Hommey [:glandium] OoO back on Apr 15]

(In reply to Chris AtLee [:catlee] from comment #1)
> I needed to add these packages for b2g emulator builds (which also require a
> 64-bit build environment):
> 
> "glibc-devel.i686", "libstdc++.i686", "zlib-devel.i686",
> "ncurses-devel.i686", "libX11-devel.i686", "mesa-libGL-devel.i686",
> "mesa-libGL-devel", "libX11-devel"

I don't think any of those -devel packages is needed for android. They are likely only required to build the emulator, which we don't build for android.

> I suspect the list would be similar for android builds. Maybe we should
> switch all builds over to 64-bit?

That's the plan for Linux builds, which needs some testing first (and bug 857697).

Comment 3

[Trevor Saunders (:tbsaunde)]

bug 857697 will almost certainly be enough it what's installed now isn't.  Its also everything from comment 1 accept ncurses-devel.

Comment 4

[Mike Hommey [:glandium] OoO back on Apr 15]

(In reply to Trevor Saunders (:tbsaunde) from comment #3)
> bug 857697 will almost certainly be enough it what's installed now isn't. 
> Its also everything from comment 1 accept ncurses-devel.

Well, technically, android builds don't use the same environment, so bug 857697 is actually not going to help.

Comment 5

[Mike Hommey [:glandium] OoO back on Apr 15]

Attachment: Try x86_64 mock environment for android builds on try

Comment 6

[Mike Hommey [:glandium] OoO back on Apr 15]

Maybe it would be desirable to use some project branch instead of try?

Comment 7

[Mike Hommey [:glandium] OoO back on Apr 15]

Comment on attachment 736692 [details] [diff] [review]
Try x86_64 mock environment for android builds on try

We'll need some packages too.

Comment 8

[Mike Hommey [:glandium] OoO back on Apr 15]

Attachment: Try x86_64 mock environment for android builds on the 'date' branch

Assuming the date project branch is really free to use.

Comment 9

[Mike Hommey [:glandium] OoO back on Apr 15]

Attachment: Try x86_64 mock environment for android builds on the 'date' branch

Comment 10

[Justin Wood (:Callek)]

Comment on attachment 736857 [details] [diff] [review]
Try x86_64 mock environment for android builds on the 'date' branch

Review of attachment 736857 [details] [diff] [review]:
-----------------------------------------------------------------

this looks good at a glance, I don't forsee any problems.

Comment 11

[Trevor Saunders (:tbsaunde)]

landed for glandium since he seems to be away and I believe wnated this in today
https://hg.mozilla.org/build/buildbot-configs/rev/932719dcdd15

Comment 12

[Aki Sasaki (not active)]

This is in production.

Comment 13

[Mike Hommey [:glandium] OoO back on Apr 15]

Attachment: Fixup

I forgot this when i changed the branch name.

Comment 14

[Mike Hommey [:glandium] OoO back on Apr 15]

Somehow, the following *doesn't* install the android* packages or java:

DEBUG: ['/usr/bin/yum', '--installroot', '/builds/mock_mozilla/mozilla-centos6-x86_64/root/', 'install', 'autoconf213', 'mozilla-python27-mercurial', 'ccache', 'android-sdk15', 'android-sdk16', 'android-ndk5', 'android-ndk8', 'zip', 'java-1.6.0-openjdk-devel', 'zlib-devel', 'glibc-static', 'openssh-clients', 'mpfr', 'wget', 'glibc.i686', 'libstdc++.i686', 'zlib.i686']

https://tbpl.mozilla.org/php/getParsedLog.php?id=21766840&full=1&branch=date

Also, B2G builds are still triggered for the date branch.

Comment 15

[Mike Hommey [:glandium] OoO back on Apr 15]

Oh, and android-debug builds are in a x86 mock env.

Comment 16

[Mike Hommey [:glandium] OoO back on Apr 15]

Attachment: Fixup, and install .i686 android-* packages

Trevor suggested that the lack of installation of the SDK and NDK may be due to the fact that the packages are for i686 only in the yum repo, and I can totally buy this. This add .i686 to the android-* packages.

Comment 17

[Justin Wood (:Callek)]

https://hg.mozilla.org/build/buildbot-configs/rev/b2a168ab6058
https://hg.mozilla.org/build/buildbot-configs/rev/2934d299a89e

Backout out for breaking try. e.g. see https://tbpl.mozilla.org/?tree=Try&rev=6ae475a235da as an example inbound push with no modifications that broke

The backout is live, when we revisit (early next week) we'll use the reserved branch that was taken for android, rather than try

Comment 18

[Mike Hommey [:glandium] OoO back on Apr 15]

(In reply to Justin Wood (:Callek) from comment #17)
> The backout is live, when we revisit (early next week) we'll use the
> reserved branch that was taken for android, rather than try

I was able to test and hit a wall anyways. The tinderbox script was trying to find a x86_64 tarball after the build. And even if that worked, that would have failed on tests.

I'm mystified that it broke try.

Comment 19

[Trevor Saunders (:tbsaunde)]

> I'm mystified that it broke try.

I was too, the best theory I can think of which I mentioned to callac on irc is that we don't explicitly ask yum to install glib2-devel.x86_64 (if I read the log right)  and it presumably gets pulled in as a dep of gtk2-devel but now we explicitly install glib2-devel.i686 and maybe yum somehow thinks that satisfies the dep and so nolonger installs glib2-devel.x86_64.

Comment 20

[Mike Hommey [:glandium] OoO back on Apr 15]

Attachment: Fixup, and install .i686 android-* packages

Refreshed against current trunk.

Comment 21

[Justin Wood (:Callek)]

Comment on attachment 737581 [details] [diff] [review]
Fixup, and install .i686 android-* packages

iirc some packages are .noarch not all have .i686 alternatives...

Comment 22

[Mike Hommey [:glandium] OoO back on Apr 15]

(In reply to Justin Wood (:Callek) from comment #21)
> Comment on attachment 737581 [details] [diff] [review]
> Fixup, and install .i686 android-* packages
> 
> iirc some packages are .noarch not all have .i686 alternatives...

afaics, there is no noarch android-* package.

Comment 23

[Mike Hommey [:glandium] OoO back on Apr 15]

Attachment: Fixups

Since we're also switching to tooltool, we don't need to fiddle with the android-* packages. This adds missing android-debug and fixes up the try thing.

Comment 24

[Justin Wood (:Callek)]

Comment on attachment 738412 [details] [diff] [review]
Fixups

Review of attachment 738412 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ +1737,2 @@
>          'glibc.i686', 'libstdc++.i686', 'zlib.i686',
>          ]

nit: lets make this:

        list(BRANCHES['date']['platforms'][platform]['mock_packages']) + \
        ['glibc.i686', 'libstdc++.i686', 'zlib.i686', ]

rather than what you have here, to resolve PEP8 violation:
E122 continuation line missing indentation or outdented.

Unless you have a different non PEP8 violation suggestion

Comment 25

[Mike Hommey [:glandium] OoO back on Apr 15]

Attachment: Fixups

How about this?

Comment 26

[Justin Wood (:Callek)]

Comment on attachment 738693 [details] [diff] [review]
Fixups

Review of attachment 738693 [details] [diff] [review]:
-----------------------------------------------------------------

WFM, no pep8 issues

Comment 27

[Justin Wood (:Callek)]

Attachment: [configs] do linux32 on date and no b2g

This does the final roundup here.

I thought there was another bug with the outstanding items where this patch belongs but I can't find it atm, the 'date' specific stuff for linux is new and I reviewed it, but when adding this in I realized we need to map from linux64 not linux32 if we're changing the mock target env here.

All passes test masters and pep8

f? to glandium as a "this is what I want" ack. (this is as I understood his prod to me in IRC)

Comment 28

[Justin Wood (:Callek)]

Comment on attachment 738805 [details] [diff] [review]
[configs] do linux32 on date and no b2g

Review of attachment 738805 [details] [diff] [review]:
-----------------------------------------------------------------

err actually I lied, apparantly I forgot my test master output here logs to files and not directly to console, this fails because linux64 is not in branch-specific platform list.

Comment 29

[Mike Hommey [:glandium] OoO back on Apr 15]

Comment on attachment 738805 [details] [diff] [review]
[configs] do linux32 on date and no b2g

Review of attachment 738805 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ +1738,4 @@
>          ]
>  
>  for platform in ['linux', 'linux-debug']:
> +    new_plat = {'linux': 'linux64', 'linux-debug': 'linux64-debug'}

You don't need this trick, the list of packages is the same for linux and linux64.

Comment 30

[Mike Hommey [:glandium] OoO back on Apr 15]

Attachment: Do linux32 on date and no b2g

Comment 31

[Chris Cooper [:coop] (he/him)]

This is in production now.

Comment 32

[Mike Hommey [:glandium] OoO back on Apr 15]

Attachment: Switch android builds to x86_64 mock environments

Comment 33

[Justin Wood (:Callek)]

Attachment: Switch android builds to x86_64 mock environments - updated for bitrot

This is the same as Mike's patch above, but adjusted for contextual bitrot.

I also slightly edited for s/branch/b/ and s/platform/plat/ to fit the assignment of mock_target onto 1 line in the pep8 80-char limit.

Of note, this can't land until we move the packages into place (myself or buildduty can do that):
* android* from the releng/ i686 repo need to move to the releng/ x64 repo.
** run createrepo after that
** wait for puppetmasters to update
* write a puppet patch to force yum cache's to flush
** land and wait for propagation to puppetmasters
** inform :dustin to also merge it to puppet320

The process is doc'd at https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Landing_Custom_Repository_Changes

In theory we should also move all i*86 packages to the x64 folder I believe now as well, but that should be coordinated elsewhere so that we can watch for fallout explicitly.

Once the packages are in place this can go live in a reconfig at buildduties discretion

Comment 34

[Mike Hommey [:glandium] OoO back on Apr 15]

(In reply to Justin Wood (:Callek) from comment #33)
> Created attachment 749227 [details] [diff] [review]
> Switch android builds to x86_64 mock environments - updated for bitrot
> 
> This is the same as Mike's patch above, but adjusted for contextual bitrot.
> 
> I also slightly edited for s/branch/b/ and s/platform/plat/ to fit the
> assignment of mock_target onto 1 line in the pep8 80-char limit.
> 
> Of note, this can't land until we move the packages into place (myself or
> buildduty can do that):
> * android* from the releng/ i686 repo need to move to the releng/ x64 repo.

You probably just want to copy them, not move them.

> In theory we should also move all i*86 packages to the x64 folder I believe
> now as well, but that should be coordinated elsewhere so that we can watch
> for fallout explicitly.

I don't think we should.

Comment 35

[Justin Wood (:Callek)]

(In reply to Mike Hommey [:glandium] from comment #34)
> (In reply to Justin Wood (:Callek) from comment #33)
> > Created attachment 749227 [details] [diff] [review]
> > Switch android builds to x86_64 mock environments - updated for bitrot
> > 
> > This is the same as Mike's patch above, but adjusted for contextual bitrot.
> > 
> > I also slightly edited for s/branch/b/ and s/platform/plat/ to fit the
> > assignment of mock_target onto 1 line in the pep8 80-char limit.
> > 
> > Of note, this can't land until we move the packages into place (myself or
> > buildduty can do that):
> > * android* from the releng/ i686 repo need to move to the releng/ x64 repo.
> 
> You probably just want to copy them, not move them.

err indeed!

> > In theory we should also move all i*86 packages to the x64 folder I believe
> > now as well, but that should be coordinated elsewhere so that we can watch
> > for fallout explicitly.
> 
> I don't think we should.

well the yum repo setup tends to have all i686 variants of its x64 varients in the x64 repo as well. In fact it expects them to be present there as well. We can do this in a deduped way if need be. I should note that I meant s/move/copy/ as well here.

Either way *that* is not for this bug

Comment 36

[u429623]

Per #releng chat with glandium, plan is:
 - hwine to do yum repo updates from comment #33 today
 - when done, those will be verified okay from date twig by glandium (Wed?)
 - assuming all okay, can land m-i/m-c after that from bug 860246 and bug 857697

Contingency plan: if concerns arise, can land to m-i first and let soak for a day or so.

Comment 37

[Mike Hommey [:glandium] OoO back on Apr 15]

(In reply to Hal Wine [:hwine] from comment #36)
>  - when done, those will be verified okay from date twig by glandium (Wed?)

It worked:
DEBUG: Installing:
DEBUG:  android-ndk5       i686   r5c-0moz3                releng-centos6-x86_64  47 M
DEBUG:  android-ndk8       i686   r8c-0moz1                releng-centos6-x86_64 157 M
DEBUG:  android-sdk15      i686   r15-0moz1                releng-centos6-x86_64 385 M
DEBUG:  android-sdk16      i686   r16-0moz2                releng-centos6-x86_64 492 M

Comment 38

[Mike Hommey [:glandium] OoO back on Apr 15]

https://hg.mozilla.org/build/buildbot-configs/rev/f68081109b89

Comment 39

[Mike Hommey [:glandium] OoO back on Apr 15]

https://hg.mozilla.org/build/buildbot-configs/rev/fbb77f1eb042

Comment 40

[u429623]

in production