virtualbox VM is not PXE booting from DRP (bootenv: sledgehammer). after tftp'ing lpxelinux.0. VM network logs (wireshark) show following error: 477 0.378296662 10.10.20.76 10.10.31.96 TFTP 159 Error Code, Code: File not found, Message: open /var/lib/dr-provision/tftpboot/pxelinux.cfg/16089a59-9abd-48c2-850a-2ac3bc134935: no such file or directory

I see that there is no such file in dr-provision file area

what can be possible bug. I might have done something.

Thanks

That is expected behavior for lpxelinux.0. see http://www.syslinux.org/wiki/index.php?title=PXELINUX#Configuration

Specifically, lpxelinux tries to fetch config files in a specific order. The first one os the DHCP client ID, which we don't use because (for servers or anything else that network boots) the client ID is a much worse unique identifier than the MAC address of the interface or the IP address the interface was assigned.

@wdennis - tip community content has a fix for ubuntu install.

You can use the current content tip with stable drp.

I also have PR in community content that moves all the partman options into the schema file. I know this will break community and haven?t pulled it in.

I also want to spend more time on it to reorg it a little more.

Thx @greg

I do think consolidating all partman directives into a single template is the sanest option... But do recognize the need to get current users on board with that change.

thanks !

@greg Confirming new partman preseed directives in `tip` community content work now...

I got: ``` #Partitioning Scheme d-i partman-auto/disk string /dev/sda d-i grub-installer/choose_bootdev select /dev/sda d-i grub-installer/bootdev string /dev/sda d-i partman-auto/method string lvm d-i partman-auto-lvm/guided_size string max d-i partman-auto-lvm/new_vg_name string testnode01 d-i partman-auto/choose_recipe select atomic d-i grub-installer/only_debian boolean true d-i partman/confirm_write_new_label boolean true d-i partman/choose_partition select finish d-i partman/confirm boolean true d-i partman/confirm_nooverwrite boolean true ``` and it successfully partitioned the drive.

Refresh my memory - the string value of `select-kickseed` is the template name WITH or WITHOU the `.tmpl` at the end?

with

Ah, that's why it didn't work :stuck_out_tongue_winking_eye:

It's the part-scheme one that doesn't want the .tmpl

is it possible to have like a bootenv with some set templates and kernel parameters, but then also have several stages with an "extra" template and kernel parameter?

and while I was typing that out, it just hit me that I guess this is a scenario I could also solve with a drpcli runner, hm...

Is there a better way to get the bare DRP server IP or hostname other than parsing out `.Env.InstallUrl` or `.ProvisionerURL`?

@lae`.ProvisionerAddress` from http://provision.readthedocs.io/en/latest/doc/arch/provision.html#rendering-templates

FYI, something is wrong in either the upstream, or more likley the ubuntu repo pointers: ``` drpcli bootenvs uploadiso ubuntu-16.04-install Error: Unable to initiate download of http://mirrors.kernel.org/ubuntu-releases/16.04/ubuntu-16.04.3-server-amd64.iso: 404 Not Found ```

this was from a new install against stable.

@rstarmer checking it ...

we had an issue w/ CentOS yanking the 7.3 ISOs with zero warnings

seems .3 ISO is gone.

also note ... if you have a copy of the ISO (Until we update the Contents) - you can install the ISO via the UX - go to the Boot ISOs menu item

or you can copy the ISO to your tftpboot/isos/ directory (either in ~/drp/drpdata, or the /var/lib/dr-provision directory)

then restart DRP

we'll put out an updated version of Contents shortly

Ubuntu has the older ISOs here too for future reference: http://old-releases.ubuntu.com/releases/xenial/ubuntu-16.04.3-server-amd64.iso

yeah ... never mind using a symlink to point "latest" at the moving target ... sigh ...

Question, or docs pointer if possible: The host I installed DR on has letsencrypt credentials, how do I tell DR who it actually is?

did you do `isolated` or production install ?

production

also, can I pass a domain name rather than IP address to the config?

@rstarmer there is a cert and key that gets installed (the self-signed). It's either in `/` or in `/var/lib/dr-provision/` directories. replace those with your certs and restart dr-provision

on the domain name - are you referring to `drpcli` commands ?

no, I was thinking about advertised address from the server, I?d rather it passes out the domain name than the IP.

was wondering if I should have passed a name with --static-ip=

via the `--static-ip` flag ?

yeah, that?s what I was wondering :slightly_smiling_face:

ah - you really don't need `static-ip` we do a bunch of magic with caching address tables and interface info - and we dynamically serve the correct IP to a client based on their network connection

the problem w/ using a hostname/domain - you have to make sure that resolves at the PXE firmware/boot level ... and any DNS issues will impact your provisioning activities

you can of course hand out DNS servers w/ the DHCP assignment ... nothing stopping you from doing that ...

ok, won?t worry there then.

but you'll be circumventing our magic logic

so the key/cert are in /, this seems like a bad place to put these items. And in reality, I?d rather point to their locations in the letsencrypt directories (so that they stay up to date). IS there a config parameter I can set somewhere?

no desire to bypass the magic!

```dr-provision --help --tls-key= The TLS Key File (default: server.key) --tls-cert= The TLS Cert File (default: server.crt)```

k. will re-provision with those.

you can specify the location ... that's just the default

so that doesn?t seem to be working. when I restart dr-provision it regenerates new self signed keys. Note I?m passing in .pem formatted key/cert, but I also don?t see any errors in trying to read them

@rstarmer I'm not sure if we accept a PEM format ... will have to check w/ @greg and/or @vlowther on that one

Ok, I?m seeing the following: ``` curl -fsSL get.rebar.digital/stable | bash -s -- --tls-key=/etc/letsencrypt/live/gitlab.kumulus.co/privkey.pem --tls-cert=/etc/letsencrypt/live/gitlab.kumulus.co/cert.pem install Overriding TLS_KEY with /etc/letsencrypt/live/gitlab.kumulus.co/privkey.pem Overriding TLS_CERT with /etc/letsencrypt/live/gitlab.kumulus.co/cert.pem 'dr-provision' service is not running, beginning install process ... Ensuring required tools are installed Installing Version stable of Digital Rebar Provision ... ```

started manually, passed the tls params, and that works.

going to restart the service and see if it takes this time

You have to add to the service file

ah

^which - where?

yeah - in `/etc/systemd/system/dr-provision.service` - assuming SystemD

The install.sh script doesn?t do anything. We need a plan for that

@rstarmer the content update w/ the 16.04.4 fix will be out later this evening the PR just needs to go through approval and release process now

ok, trying that now. I?ll let you know if/as I succeed

Yes, success, I had to update the /etc/systemd/system/dr-provision.service file: ``` [Service] ExecStart=/usr/local/bin/dr-provision --tls-key=/etc/letsencrypt/live/gitlab.kumulus.co/privkey.pem --tls-cert=/etc/letsencrypt/live/gitlab.kumulus.co/cert.pem ```

yay !

now I?m stuck, the UI just spins on loading plugins

shift-reload

helps a lot if I read the error message. gotta go find my glasses? (e.g. you must install providers before installing plugins?)

interesting behavior with the terraform plugin, only the first node gets provisioned, and the system state doesn?t seem to get updated completely, leaving the one provisioned node in ?power off? state, thought it is running in packet.

thoughts on how I might debug this?

I have an Ansible role I wrote to setup dr-provision on alpine/debian/centos? just updated it to install 3.7.0, but post setup steps seem to be failing (setup a new admin user, drop the rocketskates user, setup all the preferences and profiles and setup the boot environments). Seems to be auth related? Did something change from 3.6.0? The UI seems to work fine.

can you share the script?

has joined #community201803

hello @ced.hnyda $welcome

Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

Sure? it fails in this section. https://github.com/stanchan/ansible-role-dr-provision/blob/master/tasks/setup.yml

I flipped it back to v3.6.0 and everything works? seems to be related to the new token auth system that is in place.

@stanchan740 - I?ll check but first pass is that the drp options changed

Thanks for the tip? I?ll try looking at the options, but I believe I took all the defaults from dr-provision --help

actually, that isn?t it.

Sigh. I?m working on it now.

I hope to have a PR for you shortly.

no rush?. thanks for the help greg!

are there API docs somewhere for dr-provision? if I wanted to create a custom frontend for it?

yes, but that is that drpcli is.

$docs

$faq

$faq

FAQs are here > http://provision.readthedocs.io/en/latest/doc/faq-troubleshooting.html

That should be close. The nav should have API or something like that.

swagger?

dang it. Something broke

something else to look at.

you can hit the endpoint with /swagger-ui

it will have a graphical UI

found it in the docs :slightly_smiling_face:

thanks

This page is best: http://provision.readthedocs.io/en/tip/doc/api.html

yah? I just noticed that the v3.7.0 was just released not too long ago :slightly_smiling_face:

@stanchan the `drpcli` usage is a very very good tutor for the API - the CLI is dynamically generated from the API - so the resources closely follow the API resources ... that coupled with the Swagger-UI should give you the complete picture ...

thanks @shane

and ... v3.7.2 will be released shorty fixing a few minor issues

already out

dang @greg - you's too fast !!

how is logging handle? is there existing support for a prometheus endpoint? wanted to do something like opentracing against provisioning jobs and display it in something like jaeger or zipkin? just openly thinking :slightly_smiling_face:

umm - well - umm . that sounds cool. We don?t do that. You can grab events from a websocket stream.

or we can work with you on a plugin to push data as appropriate.

plugin to push to prometheus sounds plausible.

:thumbsup: would be interested on working on something like that

What would ?a prometheus endpoint? need?

I haven?t looked at any of it. We can off-line it as well.

that part is easy to implement? opentracing would be the interesting part

Plugin is definitely the best way, IMO ... but we also support Websocket events - so you can register for specific events via that standard method, details: http://provision.readthedocs.io/en/tip/doc/integrations/websocket.html#rs-websocket

is 3.7.2 just release? it says 7 hours ago

yes

here's a websocket listener that can log out to prometheus https://github.com/closeio/socketshark

prometheus is more for internal service metrics? distributed tracing is used for end to end transactions. all cncf projects.

still returns an error

``` 2018/03/03 13:31:16 &{403 Forbidden 403 HTTP/1.1 1 1 map[Date:[Sat, 03 Mar 2018 21:31:16 GMT] Content-Length:[0] Content-Type:[text/plain; charset=utf-8]] {} 0 [] false false map[] 0xc4201b6800 0xc4200a6370} ````

I?ll revert back to v3.6.0 for now for testing

Your playbook needs a little tweaking.

is this valid?

The problem is the password setting section of the playbook

`RS_KEY=\"admin:password\""`

I?m fixing other things too

HMM - should be, but I?m not getting the admin password set.

or should I switch to tokens

You can, but I?m almost there.

not sure why ```drpcli -U {{ provision_admin_user }} -P \"{{ provision_admin_password }}\" prefs list >/dev/null 2>&1 && exit 0 || drpcli users password {{ provision_admin_user }} \"{{ provision_admin_password }}\" && exit 99```

is not working correctly.

oh? that part isn?t fixed yet :slightly_smiling_face:

sorry

where are you getting the maperror thing?

just running `drpcli users list`

okay - well if admin?s password isn?t set then it will have problems if you set RS_KEY=admin:password

don?t need the extra quotes

ah

see the issue

`export RS_KEY="admin:password"`

it never changed the password :slightly_smiling_face:

right!!

I feel some more unit tests and a v3.7.3 coming on

Can?t set user passwords for some reason.

yah? I see the same issue

it responds back like it did something

yeah - in 3.7.0, to fix all the deadlocks in 3.6.0, we introduced system to prevent that. The password save is a special pass that we didn?t undo all the testing for.

I?ll use the default password for now? I tried using the API to change the password without any luck

yeah - the cli and the API use the same backend path.

@stanchan740 - I fixed the user bug and cut a v3.7.3 release. I have a pull request against your tree that does quite a few changes and fixes. It seems to work for me.

@greg Thanks! Looks good. Will merge. I added a few things for idempotence in my working branch. Decided to call the dr-provision API directly for that since it seems to be much more predictable.

Really enjoy using dr-provision? much better and easier to use then cobbler! It just seems to work without much fiddling around. Will be working on an awx integration to do a tensorflow on top of kubernetes demo. Looking forward to digging into the code a bit more, too.

cool

@stanchan740 I looked at API integration w/ Tower (AWX) earlier. It would be a natural plugin to push machine updates into the AWX including when machines are online or not. Something to talk about 1x1

@stanchan I'm only skimming through chat but 3.7.0 deployed fine for me with the following: ``` - name: Configure DR Provision API user shell: "drpcli users create {{ provision_api_user }}" args: creates: "/var/lib/dr-provision/digitalrebar/users/{{ provision_api_user }}.json" - name: Update DR Provision API password shell: "drpcli -U {{ provision_api_user }} -P \"{{ provision_api_password }}\" prefs list >/dev/null 2>&1 && exit 0 || drpcli users password {{ provision_api_user }} \"{{ provision_api_password }}\" && exit 114" register: provision_password_update changed_when: provision_password_update.rc == 114 failed_when: provision_password_update.rc != 114 and provision_password_update.rc != 0 - name: Update shell environment with DR Provision API credentials copy: content: "#!/bin/bash\nexport RS_KEY=\"{{ provision_api_user }}:{{ provision_api_password }}\"" dest: "/etc/profile.d/dr-provision.sh" mode: 0755 - name: Remove default rocketskates user if different api_user is set shell: "drpcli users destroy rocketskates" args: removes: "/var/lib/dr-provision/digitalrebar/users/rocketskates.json" ```

well 3.7.1

(hadn't modified that since 3.2.0 or whatever was out last august)

@lae I think that would only work if you already the admin user created and password set. A fresh install wouldn?t work. Until 3.7.3

yah? I usually create a new instance everytime I test a change

Hello all I am trying to install Centos 7 on one of our servers using our drp community content.. *It starts automated install but asks to create 1MB biosboot partition -> *Your BIOS-based system needs a special partition to boot from a GPT disk label. To continue, please create a 1MiB 'biosboot' type partition.*

so looks like we will have to modify our centos7 KS template to create biosboot partition part biosboot --fstype biosboot --size=1 {{if .ParamExists "operating-system-disk"}}--ondisk={{.Param "operating-system-disk"}}{{end}}

Any suggestions please.

And after some research I found that when exactly the biosboot partition is required - did the system boot in EFI mode or BIOS mode? - EFI - use gpt and never make biosboot - BIOS - is the disk larger than the max for msdos (2TB)? - yes - use gpt and ensure there's a biosboot partition - no - use msdos

So in our preinstall script (ks) we should be looking for the EFI capabilities and the disk size.

Please let me know your suggestions on this. Thanks.

@rakeshrhcss - I?m guessing your system has disks that are greater than 2TB in size.

My guess is for the time being, you will need to create a custom install bootenv with your own partitioning layout.

until someone can get to looking into templatizing the Centos installer to take custom partition sections.

@wdennis uploaded a file: https://rackn.slack.com/files/U416T0AAX/F9JNGVBH8/image_uploaded_from_ios.jpg

@zehicle There are multiple ways to do handle inventory with awx. Ansible 2.5, which should hit rc soon, has a few changes to how inventory can be handled in awx as well. Going to have to experiment with a few implementations. The drp dynamic inventory script works fine as an inventory provider in awx, but pushing the inventory updates to awx might be a better option.

some progress to show off... Immutable Image Deploys! https://youtu.be/tDcEzirTLbo

@zehicle You made my day

when an iso is uploaded and it shows up in the isos list, it means it?s completely uploaded? is it a blocking state?

@zehicle May I ask more info about the packer-to-drp flow ? Like which output format is used on packer side, how it's digested on DRP side ?

yes! 1) Raw Format 2) you have choices, right now, it's in the ISOs area

@stanchan740 - if you use the drpcli command, it returns upon completion.

just noticed that it?s blocking call :slightly_smiling_face: thanks!

With regard to polling from a different thread, then it will show up as `.<filename>.part` in the iso directory while it is uploading and upon completion it gets converted to `filename`

for some reason, it downloads it everytime? but it should skip if it already uploaded?

``` for _, iso := range isos { if iso == env.OS.IsoFile { return nil } } ```

no. It assumes that you are replacing updating. We could probably make it smarter, but ?

Let me check to be sure though

yah? I might just add a check for it in the isos directory

Actually, that would be a pretty good enhancement.

If the bootenv is already available, we don?t need to upload the iso for that bootenv.

it would then skip it.

yeah - I think that makes sense and would make things simpler for the installer.

the blobstore is currently just a wrapper around the filesystem? replaceable with S3 or minio in the future I assume?

Yes to the filesystem piece. In general, our philosophy is more to let you reference your files where you want.

You can build a bootenv that references or use the repos params to define the external location of the iso repos. The same with kernels and initrds. we owe some more docs and examples on this, but the point is that through the repos params, you can reference separate off DRP locations for just about everything.

You could use NFS today if you wanted to maintain the DRP managed location aspect.

cool? that makes since

Really exciting stuff!

thanks @greg

has joined #community201803

Hi all, I'm a complete noob with this lot! I'm wondering if anyone can point me in the direction of setting up Rebar to PXE boot Rock64s. I am trying to boot ayufan linux builds... Thanks in advance!

hello @rackn.slack $welcome

Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

I have a Rebar server set up and running. I am building Self Hosted Kubernetes clusters with it. It is awesome!!!! I am going to provide the modified krib template when I have it thoroughly tested. I have a 20xRock64 cluster that I'm now looking at installing it onto!

Sounds great - the Engineering folks are driving in to work right now and will be online shortly to provide a response

@rackn.slack - welcome - is this what you're referring to? http://wiki.pine64.org/index.php/ROCK64_Main_Page

...yes that's the fellow! I have 20 of them in a box! 4Gb memory makes it all possible!

Are you planning to run Digital Rebar Provision (DRP) on one of these, too - or are you just planning to install to them ?

...I thought of that but you need 6Gb minimum!?

nah - I run DRP on 768 MB VMs regularly ...

it's possible the old Digital Rebar ver2 had a 6GB memory footprint requirement, but the current DRP ver3 does NOT

...I think I tried it tentatively but stopped when I got a message saying min 6Gb. I *may* have a project that could use that though...

my first issue is installing ayufans linux on them over PXE. Once I get that cracked I'm in business!

we do build DRP (ver3) for ARM64 architecture, but it does not receive heavy testing - in general - we're a single Golang binary (only 30 MB in size), with very very few external dependencies (currently only 7zip, bsdtar, and unzip)

we do not have any patterns for you on PXE booting ARM64 architecture, we are certainly available to help here on the #community channel as questions or issues arise ... but right now, we haven't been PXE booting ARM64 hardware

Ok, thanks. I'll keep on looking.

we def. should be able to PXE boot ARM64, just not a lot of patterns for you to follow

hi there! Do you guys can explain to me why after some time on sledgehammer-wait stage I am loosing the connectivity to my servers ? They seems to lose their IP even though the DHCP lease is still valid. I tried to renew it using the dhclient ethx command, they get their IP back but still impossible to ping... The only solution I have found so far is to reboot the server...

...I have not been able to find any! I've got as far as the DHCP server returning 'a' filename for the Vendor Class, I just don't know what it should be, or where to get it!!!!!

post your details here - we have some guys that are really good with hardware - may be able to sort out what needs to be served for ARM64

@florent.wagener - are the machines in Sledgehammer when they lose their lease, or in a provisioned OS instance ?

@shane they are on sledgehammer

What I have so far is the rom loaded with ayufan's u-boot loader. This, in theory, allows the Rock64 to PXE boot. I have a MS$ DHCP server configured to send 'a' boot file name when the Vendor Class = 'U-Boot.armv8'.... but that's as far as I have got.

there are a couple of things that may be at play: * increase the Preferences setting for DHCP Lease time * `dhclient` may not be running persistently inside the sledgehammer image (I'll hav to check this) * we use Tokens to authenticate the Machine to DRP - those Tokens have a timeout as well (but theoretically, that shouldn't effect the DHCP lease)

I'll try to extend the active lease time/reservation lease time and see if there is a difference :slightly_smiling_face:

looking at my sledgehammer, it's not running `dhclient` so it's probably a single `dhclient` lease request, which would explain the issue

nothing is trying to renew the lease

extending the Lease on the DRP side will work as long as you make it really long - but this isn't necessarily the best solution - looking in to it

thanks !

on a recently rebooted machines (less than 30min) I can see the dhclient is running: ```<sledgehammer> [root@E16968917902274 ~]# ps aux | grep dhclient root 3238 0.0 0.0 113372 13140 ? Ss 15:04 0:00 dhclient eth0```

on a non recently rebboted machine the dhclient isn't running: ```<sledgehammer> [root@localhost ~]# ps aux | grep dhclient root 6865 0.0 0.0 112660 972 tty1 R+ 16:22 0:00 grep --color=auto dhclient```

Is drpcli still running?

Dhcliebt may exit when drpcli exits out of the sledgehammer service

no it isn't

on the other machine, it is: ```<sledgehammer> [root@E16968917902274 ~]# ps aux | grep drpcli root 3342 0.1 0.0 731576 17120 ? Sl 15:04 0:00 drpcli machines processjobs f8466cb9-acba-43a5-b0e8-1bcc5954b86e ```

that's weird because I have no job running on any machine at the time though...

`f8466cb9-acba-43a5-b0e8-1bcc5954b86e` is the UUID of the last job that was executed 10min ago

`drpcli` may continue to run, depending on your stage actions when the last stage runs

interesting when I try to run a drpcli command on the machine without IP, I got this error: ```Error creating sessions: CLIENT_ERROR: Get https://127.0.0.1:8092/api/v3/users/rocketskates/token: dial tcp 127.0.0.1:8092: getsockopt: connection refused```

the IP is wrong here...

if the machine has no IP - then it can't reach back out to the DRP endpoint

we provide the DRP Endpoint IP addr to the `drpcli` dynamically (if `--static-ip` is not set on the DRP side)

Got it.

makes sense :slightly_smiling_face:

So - the lesson is ... Sledgehammer brings up `dhclient` - and if the Runner (`drpcli`) isn't running, then the DHCP Lease times out ... there is also the Token expiry that the Runner (`drpcli`) is operating with, which can also time out

these things need to be managed for long living Sledgehammer runs

most of the use case patterns with Sledehammer are quick use - and move in to installed OS ... but in the case of "ready state" infrastructure - we need to harden Sledgehammer to be longer lived with it's management of Runner token and (subsequently) DHCP client lease renewal

Yep :slightly_smiling_face:

some of the can be effected with the Preferences settings by simply extending the times to really long lease lengths

@rackn.slack What type of firmware and network booting situation does those rock64 boards have?

Anyway this is not a big issue for our testing purposes, but it could be in production as we might need to have a "ready state" for our servers.

Im gonna try to extend the lease to see if that changes anything

@florent.wagener I think it is some legacy code left over from some DHCP shennanigans we had to pull for DRv2.

If so, unwinding that behaviour should be a simple thing to do.

@vlowther cool! Waiting for the fix then :slightly_smiling_face:

@vlowther ...they have nothing installed by default. You flash them with https://github.com/ayufan-rock64/linux-build/releases/download/0.6.25/u-boot-flash-spi-rock64.img.xz to get network booting going. This all seems to work nicely.... i.e. I see the required BOOTP messages using WireShark.

@rackn.slack - if you are using DRP as the dhcp server, it would be really useful for use to see a debug trace of the DHCP packet stream. This can be done by turning the DHCP Debug up to debug and catching the output of DRP.

FYI - seeing this on my Ubuntu 16.04 installs - just hangs here on this screen for a while near the end of the install...

@wdennis uploaded a file: https://rackn.slack.com/files/U416T0AAX/F9JN6UBG8/image_uploaded_from_ios.jpg

Ctl-Alt-F4 to see console messages

it's hanging in the preseed somewhere - should be a clue on screen 4

@greg...I'm not using the DRP as the DHCP server in this case. I have to use MS$. I am using WireShark however. What filter would you like on that? How do I get the reslting file to you?

just dhcp messages. @rackn.slack

There is a bigger issue. We don?t have an arm-based sledgehammer yet.

...that will slow things down!

@shane Interesting- Ctrl/Alt/F4 not working to bring up terminal...

sometimes try ALT-F4

@greg is that something that is currently being worked on in V3?

Ok @greg that worked

@wdennis uploaded a file: https://rackn.slack.com/files/U416T0AAX/F9KFPQ2PP/image_uploaded_from_ios.jpg

(Sorry for the screen reflection...) looks like it?s hauling down net-post-install.sh & running it

It has been a lower priority. We don?t have one today. @rackn.slack We don?t have hardware to play with it and we haven?t had customer interest to drive it.

@rackn.slack It is sorta driven by customer demand and/or comunity involvement.

and ww mostly want to target ARM64 stuff that is running a UEFI firmware

rather than other, less well documented network boot protocols.

We are working on a new job runner that may affect existing workloads, so I am looking for feedback on the design and implementation before we replace the current implementation with it.

https://github.com/digitalrebar/provision/blob/add-auto-reconnection-and-make-an-FSM-machine-agent/api/agent.go <-- the code for the new job runner. @wdennis and other interested parties should take a look at the comments in that code and give me feedback as to how it will affect your current workflows.

@vlowther What are motivations for change? (simplification?)

Easier maintenance and simplification.

To not have to remember the dang STOP all the time.

:slightly_smiling_face:

With this runner will do what I consider to be the Right Thing by waiting by default when there is nothing to do, rebooting whenever the bootenvs change (unless it shoudl exit instead to let an OS install finish)

I do not believe it will affect anything I?m currently doing; I only have a simple workflow (not default, which in my implementation is empty, but tied to a custom profile) and the KRIB one

This is my custom profile workflow: ``` "change-stage/map": { "discover": "prep-install:Success", "prep-install": "ubuntu-16.04-install:Reboot", "ubuntu-16.04-install": "complete-nowait:Success" } ```

And my KRIB one: ``` "change-stage/map": { "docker-install": "krib-install:Success", "finish-install": "docker-install:Success", "krib-install": "complete:Success", "runner-service": "finish-install:Stop", "ssh-access": "runner-service:Success", "ubuntu-16.04-install": "ssh-access:Success" } ```

Thank you, @wdennis - you answered a question for me.

@florent.wagener https://github.com/digitalrebar/provision-content/pull/68 <-- should be in tip soonish.

@vlowther w00t !

So @greg is the ?curtin? thing something you guys invented?

No. It is a tool in ubuntu/maas. MaaS wraps a lot of crap around it.

We drive it for now. It was quick and expedient. I want to eventually replace it.

I?ve been looking at ignition from CoreOs ne Redhat, but it has issues too.

I?ve tinkered with changing ignition and have some stuff lying around to switch to it, but it has more work to do.

I like ignition better because it is go-based and easier to work with but it doesn?t have the pieces to handle setting up bootloaders like curtin does.

Anyway, it is the way it is for right now.

@wdennis

The idea is very cool, congrats on shipping something :+1::skin-tone-2:

In other news, all my Ubuntu installs (which are the only kind I do ;) are hanging on the net-post-install.sh phase...

@wdennis uploaded a file: https://rackn.slack.com/files/U416T0AAX/F9KFKGWAF/image_uploaded_from_ios.jpg

yeah _ I saw. I tried it and it passed for me.

This wasn?t happening to me pre-3.7

hmm

can you `alt-f2`

ps -ef | grep drpcli

?Works on my machine? ;)

Trying to think about changes that could change your path.

From this work flow: ```"change-stage/map": { "discover": "prep-install:Success", "prep-install": "ubuntu-16.04-install:Reboot", "ubuntu-16.04-install": "complete-nowait:Success" }```

@wdennis uploaded a file: https://rackn.slack.com/files/U416T0AAX/F9LE1675L/image_uploaded_from_ios.jpg

I prefer to do `complete-nowait:Stop`.

We thought what you have show have worked, but we might have been sneaky and changed something.

I?ll change it as you prefer, and see what I get

@wdennis - what version are you running?

Did your machine make it to complete-nowait while it was hung? in the UX.

I think I see the change that might have changed the behavior. STOP should fix it.

@vlowther?s new stuff will also fix the problem.

The new stuff will fix it in two ways.

@greg Yes, it shows as `complete-nowait` in the UX

I just reboot the nodes, and they're fine

Also, running v3.7.1 right now - I'll upgrade soon(tm)

has joined #community201803

When does the /var/lib/dr-provision/tftpboot/machines/$UUID folder get created and the kickstart file stuck in it?

@killsudo $welcome

Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

Kickstarts and all templated files are never rendered to disk - they're rendered on the fly by DRP

and served from the in-memory virtual filesystem

I've got my DR-Provision setup and working with some test vm's. I can get an ip via dhcp from rebar and see it load ipxe then sledgehammer

but after the machine is registred and I change it's bootenv to centos7 and set the ks template to centos-7.ks.tmpl and reboot the test vm centos7 never installs

@shane uploaded a file: https://rackn.slack.com/files/U6QFVRJNB/F9KNPLMTN/view_kickstart_or_preseed.m

are you familiar with the `drpcli` command ?

interesting.. this very much like VMwares AutoDeploy for vCenter for automating ESXi

yea i've been playing drpcli and getting comfortable with it

(except MUCH MUCH better ....) :slightly_smiling_face:

My end goal is to wrap up the drp API for StackStorm

can you please provide `drpcli prefs list` output

@killsudo uploaded a file: https://rackn.slack.com/files/U9KSXT6CV/F9K76CKG9/-.txt

if you have the defaultStage setting set - then you need to manipulate what machines are going to be installed with via the use of Stages (workflow), and not BootEnvs

hmm yea I clicked that wizard button and was just trying to find better doc's on how to use that workflow tool

since you have defaultStage set, you need to set a Stage on the machine, which contains a BootEnv definition - do not manipulate the Machine via BootEnv settings

we don't have the workflow fully documented yet

ok so Im not crazy, my research ended up at http://provision.readthedocs.io/en/stable/doc/workflows.html

for Docs - please switch to `tip` version - they are MUCH MUCH more updated

which while nice, doesn't help that much if you are already very familiar with dhcp/pxe/autodeploy etc

(use the lower right floating version selector)

so I guess workflows are optional then?

my use case might be slightly different then a normal user

"technically" workflow (stages) can be optional, but we have found everyone adopts them once they understand them

`defaultBootEnv` should be sledgehammer to be consistent.

I don't want/need a TheForeman/Cobbler etc experience with lifecycle management. When I start a provisioning run I have already acquired all of my varibles from different inventories etc so I mostly need a nice API driven way to define a machine and when it boots just pull down the os with a firstboot script etc. Once I get into the OS to pre-stage some stuff I have to back out of the OS cleanly and hand off the machine to a third party

so no leaving keys etc and zero need to be able to come back to it in the future

fire and forget style setups

ideally if I can tie a 'machine' to a dhcp option 82 string, then assign a bootenv to get the SO on disk then the better

then I don't even need to futz with subnets or mac's

I think you'll still find that Stages and workflow is useful for you. Particularly if you're "first boot script" needs to vary between machines.

considering your request, the stages/runner would do exactly what you are asking for post provision. No keys, no ssh, no return access

You can build the workflow to do common boot things, select the OS, then do the post-provisioning twiddly bits. Our install method is good because our agent disolves by default after provisioning - and you don't need to leave artifacts behind

I already have my OS's 100% handled in Ansible beautifully including backing myself out gracefully. So mostly just need winrm with https on windows and ssh on linux/esxi so stackstorm can launch my ansible/powershell plays

so this is sounding good

so Default BootEnv in global properties is now sledgehammer, thanks @greg

ok so cleared out my workflows

so should I still use the wizard or start by building a simple single line flow

the wizard builds a basic one

@killsudo - let?s be clear on what you are trying to do.

You already have inventoried, IPMI managed, Raid configured, Bios adjusted system that you control from another system or systems.

+1 - http://provision.readthedocs.io/en/tip/doc/arch/provision.html tip has much better write up and starting to make some sense

@greg yup lets assume that

You want the to PXE boot these machines (triggered externally) and the result of some intervening process is a booted OS (of some type) with an ssh key (on winrm enablement).

I assume you also probably want some notification that this is done.

notification would be a nice extra

I can listen for or poll events (stackstorm uses OpenStack Mistral under the hood for workflows)

How do you expect to inform the black box of the OS selection and ssh key to use?

I assume the ssh key will be consistent and constant to your secondary provisioner.

depends on the logic once I start manipulating it

but yea the key could be static to start with and just a simple edit to the kickstart

not hard for me to have the OS call my stackstorm hook to notify it that it's alive and send along some info

I plan to put the mistral workflow into pause state while I wait around for the boot/dhcp/pxe/os load part

then I can resume when I see the incoming call

okay - now we are getting some where. A `task` that calls home when ready.

How do you want to install the OS?

you mean from where or do I want to use disk images?

well - kickstarts/preseeds, raw images, rootfs images?

immutable in-memory CoreOS/Rancher?

probably more then one, I have access to the mac addresses and know the OS before talking to drp

ips?

yup I can also know that

or I can just use my option-82 string that my dhcp relays are injecting

and map that to an OS

yep - both are options. Requiring different paths.

yea that's where I am stuck is trying to understand all of my options

Are you using DRP as a DHCP server?

I do have a working dhcp/pxe system now, it's just old school centos/isc-dhcp/pxe with some bash and ansible automating it

I can if that's best

but no issue using relays and next-boot / next-server options in regular dhcp

Do the machines have sane IPXE clients?

does that exist?

:slightly_smiling_face: fair enough.

http bzimage support

yup, all of my current systems chainload ipxe

or they can if that's what your getting at

it's all enterprise kit

yeah.

more concerned about uefi stuff (argh)

well there is uefi ipxe bootloader

if it works

I know Hyper-v Gen2 vm's will be a pain with uefi being enforced

VM?s uefi suck

I think my first test run with drp, those failed to even reach ipxe

my other test machines seemed happy with the defaults

They all seem to assume a disk

To reduce your initial learning space, you may want to use your existing DHCP services to chain load into sledgehammer/discovery.

well I have my test lab already successfully working with DRP dhcp service

Okay - well - I was more thinking for your IP management system that you already have, but either way.

I?d let the machines get discovered. Default to the `discover` stage.

Transition that to a new stage with a new task that acts as an in-line classifier.

what is an 'in-line classifier'? The template system?

parse your option 82 and convert to an install stage. (centos-7-install or whatever). Transition that install stage to a stage that installs your post reboot hook and reboot the machine.

`in-line classifier` == stage with a task represented by a template that would parse the option 82 out of the dhcp lease file and convert it to an install stage.

lets start simpler so I can build up to that. I need to get my head wrapped around the basic 1.2.3 operations. otherwise my questions are not gonna be as helpful or concise

dhcp > pxe > discovery > centos7

whats the workflow way to just discover every machine that network boots against drp and install centos7 automatically with the default ks

set the default stage to centos-7-install

what about if I want to hit 'discover' first then centos7?

that is where you will need a workflow

ok so on that page and loaded up the default global workflow with the wizard

Read about workflows and `ssh-access`.

You aren?t using virtualbox

Will want this:

discover->centos-7-install:Reboot

centos-7-install->finish-install:Stop

ok so seeing that

would 'discover->sledgehammer:reboot|centos7-install:reboot->finish-install:Stop' also be acceptable?

we don?t have a sledgehammer stage.

*sledgehammer-wait*

sledeghammer-wait is just a holding stage. It is meant as near-line waiting system.

ok, so that's more of a live environment I could break into and do actions on the box before proceeding with the OS install

like updating firmware or something if I was to write that task up

or heaven forbid automate them. :slightly_smiling_face:

as Stages !

RackN has stages that do that and more.

Sledgehammer is pretty rich so you could use stackstorm to trigger things there if you wanted.

yea your previous statement cleared it up, I was confused as to what the actual difference between discovery and sledgehammer was

Discover will put ssh keys in place if you define enough parameters.

I was thinking sledgehammer was like the autodeploy boot env that loads and send the ipxe details back to vcenter and must be booted the first time thru on an unknown machine

it can be used that way, but not required.

You could direct create machines and just jump to centos install

That was another path

well if guys like the workflows I'll give them a chance

most people want a lifecycle around their machines. Workflows really help with that.

hmm still no go, still failed to fetch ks

and $url:8091/machines/8bc9b109-0034-42d6-847b-91d716d65333/seed just 404's

Kickstart is rendered as `compute.ks` (not seed)

for centos-7-install ... for an ubuntu-16.04-install you'll get `seed`

there we go

so it did generated the ks, and this time my vm went thru discovery then auto-rebooted and pxe booted into centos7

so progress

the machine details also show it inheriting the centos7 bootenv from the workflow

weird.. 'curl: (23) Failed writing body (8520 != 16384)'

Depending on file and timing, it can change.

the kickstart/preseed is only available to be rendered during a provisioning activity

I actually create a "phantom" machine I can place in to various BootEnvs to render templates against to test template rendering

that curl was from dracut inside the testvm when it tried to locate ks

apparently 1gb of ram isn't enough with the fs

nope

Correct

not for CentOS 7 - you need 1.5 GB

(that's what I use on my test VMs)

sledgehammer is rich

Ubuntu is ok w/ 1 GB

yup you guys are on it, set 4gb and now centos7 is running KS

I usually use 2GB. It is designed for real servers.

yea not an issue, I can change those values on vm's before and after the install

I assume windows and esxi are supported well enough if your willing to put in the grunt work?

both are RackN commercial components ...

looked like the explode script knew how to handle those isos

what does that mean? it'll never work without some addon?

we support Windows via Image based deployment (eg Immutable Infrastructure)

there are no community based Open Source pieces to do ESXi or Windows

but I can someone implement the details themselves if they want to?

if all you want is to get pxe to launch the winpe stuff and image the disk?

it should be possible with the right effort

I'll look into the RackN stuff, I assume questions regarding esxi/windows isn't really for this #community channel then?

is it only windows that has a commercial piece?

Both Windows and ESXi - both have required a lot of engineering work to get working repeatable and at scale.

yea I bet, both gave me a headache on my current pxe system

heyo, first box is online. not to bad now that Im grasping the steps

odd one with this Hyper-V Gen2 vm.. I can see that DRP serves it up the proper efi boot file via option 67 and then ipxe loads but after ipxe tries to fetch '' it bombs out

dr-provision2018/03/07 07:24:38.728050 [3271:665]static [error]: /home/travis/gopath/src/github.com/digitalrebar/provision/midlayer/tftp.go:82

dr-provision[3877]: [3271:665]TFTP: ipxe.efi: transfer error: sending block 0: code=8, error: User aborted the transfer

@killsudo - check in the tftpboot directory for that file. It should be there.

@killsudo What was the error on the ipxe side?

Our pxe serving logic for UEFI has been built around "whatever recent-ish tianocore expects", and I have no idea if hyperv uses a tianocore based UEFI stack or if it rolls its own.

and that "User aborted the transfer" error you are seeing is usually just the firmware or ipxe getting the size if the thing it wants to load next.

- quick poll ... would you like to see Immutable Provisioning demo on the next meetup? (Immutable = image based deployments). Please vote on the meetup page: https://www.meetup.com/digitalrebar/polls/1263248/

following up on the DHCP issue I mentioned yesterday (even though I know that you have fixed it in the last tip version). Extending the lease seems to do the trick !

Yep - it's a band-aid though ... and not necessarily the right solution for some environments that want short renewal cycles on their leases ... but, glad that works for the moment :slightly_smiling_face:

we'll have this fix in a release out soon ... or you can update to `tip` to get it

Im working on something else right now but I will definetely test it :slightly_smiling_face:

@vlowther I think it might be related to this - http://git.ipxe.org/ipxe.git/commitdiff/9366578

Possible. We pull in the latest ipxe.efi whenever we cut a build, so given the patch date that fix should be in there.

Doesn't mean that patch hasn't been reverted or broken by something else in the meantime, though. :confused:

yea looks like hyper-v is gonna have to remain a virt disk copy as that comment doesn't sound reassuring until I can revisit it

I usually try and start stuff with hyper-v as an edge case cause if it works in hyper-v I can pretty much make it work on anything else

That's kind of like saying "I hit my self in the face with a shovel before digging a hole, because I like the feeling it gives me ... "

yeah, I go with something a little more mainstream, like qemu+kvm

:slightly_smiling_face:

heh it's what you live with when hyper-v is involved. Now I get to finally move on to testing proxmox and vmware. Both of their networking layers are virtualizied. proxmox is openvswitch with vlans dumping into a evpn/vxlan fabric. vsphere is the same with some nsx involved and dumping into a evpn/vxlan fabric. All subnet gateways are anycasted gateways(using evpn) on juniper mx routers

This topology works awesome right now with the pxe booting of physical servers but the virt side throws a few extra hops in the way that *should work*

@killsudo - if you already have DHCP relays working in those environments, then it should work pretty cleanly.

yea it's more making sure ovs or nsx doesn't eat the broadcast

@killsudo uploaded a file: https://rackn.slack.com/files/U9KSXT6CV/F9MBHMQ06/-.txt

Looks like this *should* be doable on Gen2 hyper-v

@killsudo - if you want to try different `ipxe.efi`, you can just place them in the tftpboot directory. If you want them to persist over drp restarts, you need to place them in the `replace` directory as well.

If you are in `isolated` mode, these are in the directory you started drp from.

If you are in `production` mode, these are in `/var/lib/dr-provision`.

If you find one that works, lets know where you got it from. :slightly_smiling_face:

Hey all. I'm trying to create a bootable iPXE image that I can use to manually point a machine at a dr-provision host on another subnet. I'm doing this because I can't run a DHCP server on the subnet the host to be discovered lives on.

Oh and for extra complications, it's also a UEFI boot only system.

I have the image booting and am trying to chain boot bootx64.efi. But I get the error ```no config file found in forcing interactive mode due to config file error(s) ELILO boot: .............```

Then just an endless string of dots.

It never seems to get any farther.

So it downloads bootx64.efi from the dr-prov server, but can't go any farther apparently.

@dave.parker - There is ProxyDHCP for this case if needed. It can in parallel only had out boot options.

@dave.parker uploaded a file: https://rackn.slack.com/files/U9BU09D51/F9N2X4NG7/menu_ipxe.txt

@dave.parker - Second, I think we?ve dropped support fro bootx64.efi because it was buggy. And switched to `ipxe.efi` because it worked better. @vlowther will have better info.

Ok.

I can't install anything in this subnet really. So I don't know that I can do the ProxyDHCP either.

okay - so - how do you control what it is going to boot the first time?

With a bootable iso, with an iPXE executable that runs that script I posted.

okay - so you could try using ipxe.efi with a pxe script that points to `http://<ip>:8091/default.ipxe`

where `<ip>` is your DRP endpoint.

yeah, that. :slightly_smiling_face:

and `8091` is your HTTP port on the DRP endpoint.

Got it

The `default.ipxe` file is rendered with enough to handle discovery and directed boot operations.

It is provided by the `discovery` bootenv. It looks like this:

@greg uploaded a file: https://rackn.slack.com/files/U02DGQYK1/F9M1H2DLJ/-.md

The loopback addresses are just because it is how I asked for it.

It basically looks for ipxe files on the DRP endpoint for the machine specifically (`<ip>.ipxe` and `<mac>.ipxe`). Bootenvs are supposed to provide one of those files for IPXE boots.

If they don?t exist, then it boots sledgehammer and starts discovery.

Ok, cool.

I'll give that a try.

Hey, that kind of worked.

It grabbed sledgehammer and booted into it. But it then set the IP address to 0.0.0.0 and is just looping `Sending discover...` over and over

sledgehammer expects DNS domain in the DHCP options, I think.

You can log into sledgehammer from the console with root/rebar1.

To see what DHCP client is getting upset about.

That maybe where you are.

I can't get in because it just keeps trying to send discover. I never get a login prompt

So the sledgehammer image is always going to try configure the network with DHCP? So if I can't have a DHCP server on the network (or there's one there I don't control) will sledgehammer not work?

It needs DHCP and what you have should work. They are a couple of things going on.

We made a change to let dhcp and drpcli run separately in sledgehammer. This is good, but may be preventing drpcli from running. It is a new change. Probably a bug.

The second is that our dhcp server requires 1 options (I think). Dns Domain.

We can look at removing that requirement.

We are just a little busy with some other things at the moment.

I don't have any subnets configured on my dr-provision server currently because I'm testing these remote installs. Do I need one configured anyway?

No. It is should be fine.

There are bugs for your use case. It would be nice to know what options your dhcp server is sending

I don't even see anything on the server end. Nothing in the logs at all.

Oh well I'm going to come back to this tomorrow.

has joined #community201803

Welcome @zerocarbthirty $

sorry, $Welcome

Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

Are there shims or example entrypoints for DRP that show things like: "This is where you put your script that SSH's into a switch and configures ip dhcp-helper on an interface so that a machine will pxeboot when it reboots" or "this is the URL where DRP can retrieve information about the network interfaces for a machine" or are defining all of the tasks left to the user?

Thanks for the question, I am on our short morning meeting with engineering, they will be checking this channel shortly.

@zerocarbthirty We provide several common tasks as part of the community content

For example, getting the information about network interfaces for a machine is done as part of initial system discovery, which runs the gohai task.

We don't have one that configured the DHCP helper on a switch -- our usual assumption is that everything is using DHCP all the time anyways, and that the networking guys don't want us touching their switches.

@zerocarbthirty - If you have the admin access to the switches in question - and if you have existing tooling to be able to implement those changes, you can build a Stage that would make switch port changes as part of the Workflow aspects of Digital Rebar Provision

More generally, we expect that each environment will need some customized content (tasks, stages, etc) and workflows to accomplish your deployment goals.

however, as @vlowther states, we don't have precanned components in the community tooling that does that - there is a bit of a chicken-and-egg issue, if you are not DHCPing your host against DRP, you'd need to build Reservations for the machine in advance, and add a machine in advance of it booting against us - and your first Stage would be to configure the Switch port to dhcp relay - then power on

you either have to have DHCP and our Discovery mechanisms to add machines in to DRP - or you have to build the information in advance to manage the Machine - then make the switch port changes and boot the machine for provisioning

has joined #community201803

@mark.yeun $welcome

Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

howdy, very cool stuff you're doing :slightly_smiling_face:

thanks !

i hope you don't mind me just popping in to ask questions...

Well, since you ask nicxely. :slightly_smiling_face:

:slightly_smiling_face: I've got dr-provision working nicely in a libvirt environment. I'm trying to get it working on metal. I have the server set up, dhcp relay is working. I have a serial console to my baremetal box. When I pxeboot, I see on tcpdump that dhcp works, and tftp for lpxelinux.0 works.

then... nothing

I believe the next thing that should happen is another tftp for a series of files, which should fail, then a tftp for pxelinux.cfg/default

fw or iptables blocking ports 8091 and 8092 on the DRP endpoint ?

wide open

Yep, that is what you should see.

so my theory is lpxelinux.0 doesn't like my hardware?

What version of dr-provision are you running, what hardware are you testing, and is it booting via UEFI or legacy BIOS?

...and what does your serial console on your Machine show ?

i _think_ it's legacy BIOS.

``` $ dr-provision --version dr-provision2018/03/09 17:45:45.087689 Version: v3.7.3-tip-5-eb82a0429c7c94bb1885cc32528c15e376417138 ```

Cool.

and I don't have access to vga -- only serial console

No worries.

vga is only for winders ... a real OS only needs a serial console ...

heh

lol nice

do you want to see the subnet def?

nothing special

What does the subnet definition looklike?

yes. :slightly_smiling_face:

haha ```{ "ActiveEnd": "10.10.24.171", "ActiveLeaseTime": 60, "ActiveStart": "10.10.24.151", "Available": true, "Description": "", "Enabled": true, "Errors": [], "Meta": {}, "Name": "kube1_subnet", "NextServer": "", "OnlyReservations": false, "Options": [ { "Code": 1, "Value": "255.255.254.0" }, { "Code": 3, "Value": "10.10.24.1" }, { "Code": 6, "Value": "10.40.20.101,10.40.20.102" }, { "Code": 15, "Value": "http://tower-research.com" }, { "Code": 28, "Value": "10.10.25.255" } ], "Pickers": [ "hint", "nextFree", "mostExpired" ], "Proxy": false, "ReadOnly": false, "ReservedLeaseTime": 7200, "Strategy": "MAC", "Subnet": "10.10.24.0/23", "Unmanaged": false, "Validated": true } ```

That is fine for 3.7.3

I wanted to make sure it wasn't forcing the wrong bootloader or something like that.

Can you throw a screenshot of the failed boot into the channel?

it's blank

...

serial

so F12, then blank

so the serial console shows the nic firmware doing its thing, loads lpxelinux.0, then goes blank?

^^^

yessir

you need to set serial console on the DRP side

actually, as usual, my bios is blanking the screen after POST

hm

I've set serial console, and I see it in the rendered pxelinux.cfg/default

but i'm not getting that far

can whatever you are serial cnsoling into the system cwith capture the output?

ah

Yeah, I suspect it is what shane mentioned then.

lpxelinux.0 doesn't download the config file, so it doesn't know to output on serial

We should make having a serial console enabled the default one if these fine days.

and even if it did, the rendered pxelinux.cfg/default has console=... for the kernel but not for itself

```$ tftp REDACTEDHOSTNAME -c get pxelinux.cfg/default $ cat default DEFAULT discovery PROMPT 0 TIMEOUT 10 LABEL discovery KERNEL sledgehammer/9743e672ff33179cd5218d8fe506c03cf2a31d18/vmlinuz0 INITRD sledgehammer/9743e672ff33179cd5218d8fe506c03cf2a31d18/stage1.img APPEND rootflags=loop root=live:/sledgehammer.iso rootfstype=auto ro liveimg rd_NO_LUKS rd_NO_MD rd_NO_DM provisioner.web=http://10.40.20.30:8091 -- console=ttyS1,115200n8 IPAPPEND 2 ```

Another thing to try would be to set option 67 in the subnet to `ipxe.pxe` and see if that works.

is your serial console *actually* on ttyS1 ? not ttyS0 ?

ttyS1 is kinda non-standard - it's what http://packet.net baremetal systems use by default

tried both :slightly_smiling_face:

i tried this, no change in behavior `drpcli subnets set kube1_subnet option 67 to '{{if (eq (index . 77) "iPXE") }}default.ipxe{{else if (eq (index . 93) "0")}}lpxelinux.0{{else}}bootx64.efi{{end}}'`

wow, that old string is still out there.

try {{if (eq (index . 77) "iPXE") }}default.ipxe{{else if (eq (index . 93) "0")}}ipxe.pxe{{else}}ipxe.efi{{end}}

are you using a USB to Serial converter, or an actual 9-pin serial port ?

from http://provision.readthedocs.io/en/latest/doc/faq-troubleshooting.html#uefi-boot-support-option-67

actually i have an avocent

um ... that's my fault ...

ok trying that option67 value

``` { "Code": 67, "Value": "try {{if (eq (index . 77) \"iPXE\") }}default.ipxe{{else if (eq (index . 93) \"0\")}}ipxe.pxe{{else}}ipxe.efi{{end}}" } ``` gonna give it a go

too much

get rid of try

there is no try. :slightly_smiling_face:

lol

do or do not

What backend system does DRP use to store information gathered with gohai/etc ?

This command: `drpcli subnets set kube1_subnet option 67 to '{{if (eq (index . 77) "iPXE") }}default.ipxe{{else if (eq (index . 93) "0")}}ipxe.pxe{{else}}ipxe.efi{{end}}'`

It uses a file-based data store.

@zerocarbthirty

Oh, I figured it was using Collins or something like that.

@zerocarbthirty - we try to keep it minimal / lightweight for embedding in things.

Nope. We want complete standalone operation + a zero-dependency install.

We can use different backend but they need to be more blob store like than Collins.

so the dr-provison binary embeds all the things it must have to install.

So I asked the question earlier about having DRP configure switchports for DHCP when it needs to pxeboot a system rather than having DHCP always on. I've been watching alot of the videos about DRP (that use packet as infrastructure) and it seems a little odd that a network of that size would want every port forwarding DHCP traffic from every host to the DHCP server all the time. Although I guess they probably don't use DR for their own DHCP/pxe so maybe they do actually configure the ports.

yes, they configure switch ports dynamically for every single host - every single host by default is isolated in a /31 size L3 boundary and VXVLAN separated (updated my typo to /31 boundary)

Digital Rebar Provision can configure ports just like they do ... it would be part of the Stages of workflow that you'd write for content

lol i thought you were making a yoda joke but I get it now. removed the "try ", retrying

the open source Digital Rebar Provision out-of-the-box does not have switch management built in

boys i see many packets

@shane uploaded a file: https://rackn.slack.com/files/U6QFVRJNB/F9MKL36H3/screen_shot_2018-03-09_at_10.10.59.png

Hmm I was under the impression that they didn't actually use any L2 VLAN/VXLAN stuff so that people can use their own overlays.

but it's amusing to me that they use /31s just because of NANOG's history of arguing about whether or not anyone should ever use /31s haha

i forgot to put serial console back on ttyS0, but the machine showed up in drpcli machines list

will give it another crack with proper serial

but you are the _man_

seems I'd have been stuck without the slack inquiry

thanks guys! much appreciated.

What type of hardware is it? @mark.yeun

supermicro X10 series with mellanox 10g

hmm - the mellanox drivers may not be in the open source centos/ubuntu - have you checked that ?

lpxelinux.0 might not like the mellanox. `ipxe.pxe` is ?newer?

I know they were a problem (especially Ubuntu 14.x)

which Mellanox cards ?

Anyway, earlier I wasn't asking about whether DRP has the ability to configure networking equipment I was asking whether there were example workflows that illustrate where one might just pop in their own code to look stuff up from IPAM/configure switches, etc

02:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

should be good - that's an older gen card - which I think is supported in the mlx4 driver

(it might be in the mlx5 driver - but both are in CentOS7 by default ... pretty sure ubuntu 16 too)

one more quick question -- I haven't looked into this yet. is IPMI support behind the pay wall?

Yep.

Hmm, I think crowbar used to manage ipmi

odd that they would decide to make that a paid for feature

We have to have some encouragement to move to revenue generating customers.

okay, if this sexy thing keeps looking sexy i'll hit up your sales team for at least ballpark pricing :slightly_smiling_face:

You can play with it in the content bundles of IPMI.

@mark.yeun - that is amazingly easy. Seeing as you?ve talked with over half the team. :slightly_smiling_face:

okay that triggers one more question. what sort of features do we get with the paywall bios support?

Dell firmware updates and BIOS config are there presently

Time to feed the :bear:!

seeing as how that is what we have the most experience.

@mark.yeun You should be able to just boot into a thin linux environment like Alpine and then use whatever vendor's tools to configure your BIOS by just having alpine curl a script that builds the config based on either the MAC or IP of the client.

omconfig is especially easy to do that with

I have implemented Supermicro support (via sum) for DRv2 and an earlier customer, and have been waiting for demand to port that support over to dr-provision.

@zerocarbthirty thanks, we have such stuff for our old heavy OS builds. lots of witchcraft though with our mixed hardcware platforms

I don't know how far along redfish is but some of that may be standardized now

@zerocarbthirty - or you could use our tools and workflows to automate and orchestrate. Or build a task/stage that runs your own tools as part of the process. DRP is flexible.

ya, our BIOS and RAID plugins go through an in-house tool that implements a standard format and idempotent config flow on top of vendor-provided tooling.

@zerocarbthirty...it all sounds so _easy_ on paper .... :stuck_out_tongue_winking_eye:

yeah, we take care of blundering into and padding all the sharp bits so you don't have to.

and i'm guessing that also Windows is paywalled too?

so you can (for instance) hand us a JSON blob representing a RAID config and we take care of driving megacli/storcli/whatever to make that config happen.

Ditto for BIOS config.

You mean in the event that we have access to the rackn stuff

@zerocarbthirty - yes, image-based deploys are paywalled. Because they are tricky usually require consulting.

oh, we have it down to where we boot into PE and use command-line DISM to do phase 1 then the system reboots and finishes the install. We didn't find it very complicated even using pxelinux

Okay - then that is good for you. We?ve found that building images and deploying them is much faster and our customers like that workflow better. It fits into CICD pipelines better and gives better controls.

I assume you are referring to non .wim images?

Correct - though I have prototypes that deploy those as well.

So you are referring to images that you drop onto the disk when the install is past what is commonly referred to as the 'PE' phase?

hey I want to thank you guys again. will go have a play, and may come back for more magic.

@zerocarbthirty: yes.

@mark.yeun: No problemo.

That is interesting, I haven't really looked for different imaging formats than .wim since we used ghost+floppies to do installs but I suppose if you include the right drivers or only use hardware that has native drivers in $target_windows_version that method could be a bit faster.

It is extremely fast - only requires one reboot of the machine to provision to a completed OS, and we support adding in post-provisioning bootstrap config changes as well

along with our Agent, which you can enable longer term lifecycle management (if desired, by default our Agent "dissolves" after initial provisioning" )

coupled with a CI/CD pipeline to validate/test your "gold" image, you can roll out patch updates very very quickly across very large scale infrastructure

you also can roll forward/roll back images quickly through this mechanism in the event a new image exhibits behavior problems "in the wild"

So, pardon me if I am nosy but you would take the image when it is at the "Please wait while we are setting everything up for you" phase which I believe they still call OOBE and then have an execution configured to grab a script in powershell or (whatever) to execute the changes?

I suppose one could also just roll cloud-init into an imagine for physical hosts

err image

Hmmm. Maybe. :grinning:

although you would want to avoid joining the domain with such a host until after the computer name is changed for obvious reasons

If that is a concern. Most definitely

Discovery uses open sledgehammer? does that do cdp/lldp to gather network info?

lldp is in the Sledgehammer image so you can definitely build a Stage to collect switch port info

Cool, i'm having the team install 40 PowerEdge 440s to play with DRP next week so I'm just trying to think of everything I am going to wonder about ahead of time.

Stages can be used to integrate in to external DCIM/Asset Mgmt/Cfg Mgmt databases as well - we can either push inventory info to them, or pull info to build provisioning decisions against

(and IPAM as well)

The problem I find mostly is that the team will rack 500 servers and then make a bunch of mistakes describing them whether it's the actual specs, the network info, whatever. so thats really the part that is the biggest pain in the ass. I didn't realize this until today but Packet never changing their hardware after it is installed makes alot of things a whole lot easier.

yep, I know that pain ...

correlating the physical infrastructure design from what it _should_ be to what it really is (or isn't; if it's broken) ... can be a mess

you can audit via Sledgehammer what the reality of your network ports are and compare that to a "it should be this" design

we use the LLDPD implementation which supports LLDP, CDP, EDP, SONMP, and FDP protocols

Well, if you devote a portion of your DC to being totally 'fixed' i.e. you won't change the specs/wont change the network, etc it makes alot of that trivial

but if you are constantly adding/removing drives+ram+pci-e cards, etc

your inventory is going to get out of date pretty quickly

one could implement use of our Agent (i.e. not let it dissolve), and built in `gohai` inventory to report back to your provisioning service or other external services on a periodic basis

then you can support continual sweeping of inventory management - these are some of the larger lifecycle management solutions that can be built around the Agent if desired

my initial thought on that was to have a maintenance mode that would cause it to boot back into the discovery to update the inventory but then I just figured out that we could probably just make a certain percentage of stuff not changeable

Sledgehammer is designed to support that use, and DRP is designed to let you in-memory/live boot systems "do stuff", then boot them back to their installed OS easily enough

but that's a fairly disruptive pattern to have ...

I always find it humorous that if you add RAM to a dell server it takes a new inventory but there is no way to tell it to send that information anywhere

Time to feed the :bear:!

use of the Agent reporting back to DRP can centralize and manage that information as opposed to the rebooting - our Agent is crosscompiled for Arm, Intel, 32, 64bit, Linux, Windows, and Darwin (Mac) currently - pretty easy to port to other things if desired

Oh, yeah man but it's not like you are going to upgrade RAM on a server while it's running

I would call those workflows. :grinning:

so there will be some amount of disruption anyway

and if it's something that can't be disrupted it should be running on block storage that is accessible from more than a single host anyway

that is a good modern design pattern, but sadly, not all shops are there ...

Yeah earlier I was moslty just asking if there were example workflows that are just missing the rackn pieces

like "this would be where rackn did this cool thing"

that would make it easy to figure out what pieces we have/need to build/want to buy

You guys have piqued my interest on this windows image thing, must find file format :smiley:

this Tuesday we will be demo'ing our Image Deployment capabilities - including the Windows Image deploy on the weekly Meetup

@zerocarbthirty - The default community content has stages and tasks in it. You can see the tasks / stages in the RackN content that is available by logging into the RackN SaaS.

When you create an account, you'll have access to some licensed content on a trial basis from the catalog.

The account turns on "registration wall" UX features.

little bonus features.... added live Task view to Bulk Edit

@zehicle uploaded a file: https://rackn.slack.com/files/U02DHRR2L/F9MMKKU04/image.png

It will be in the latest after we test it a bit

has joined #community201803

$Welcome

Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

hey guys, quick question, I have 3 burnin jobs that are stuck in a "running" state and I can't delete them. How can I force them into a fail state ?

I've tried this: `drpcli jobs destroy ce9c668a-7888-4308-84c6-e1591deedc0d --force` without success :disappointed:

nevermind I find it: `drpcli jobs update ce9c668a-7888-4308-84c6-e1591deedc0d '{"State": "failed"}'`

am trying to setup kubernetes cluster on virtualbox VMs (3 VMs)

have completed kubernetes setup via kubespray playbooks

however, unable to access the dashboard

it says 'unauthorized'. Any idea how to fix it ?

Thank you !

``` { "kind": "Status", "apiVersion": "v1", "metadata": { }, "status": "Failure", "message": "Unauthorized", "reason": "Unauthorized", "code": 401 } ```

@amit.handa - I think you need some kubeproxy magic, but not sure. Something like this may help you start. https://github.com/kubernetes/dashboard/issues/692

thanks greg. Let me look for cure

@amit.handa are you trying to access via the `kubectl proxy` command ? http://provision.readthedocs.io/en/tip/doc/integrations/krib.html#kubernetes-dashboard-via-proxy

I am opening it via https://<kube-master-ip>:6443

as mentioned in the drp docs

has joined #community201803

@amit.handa if kubespray changed security defaults then the docs will be out of date. The integration just supplies the inventory. It's likely they tweaked the auth system

@wayneeseguin $welcome

Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

thanks @zehicle for the info, I am new to kubernetes. ll check and update.

thanks @greg

I stopped the docker proxy container and ran following on master node

```kubectl proxy --address 0.0.0.0 --accept-hosts '.*'```

I can see the dashboard

has joined #community201803

I'm exploring use of digital rebar to provision bare metal machines in the office and in the data center. Is it a lot of effort use digital rebar when the local network already has a dhcp server? My experience is more in devops and software provisioning rather than traditional ops

@jpresley there are several ways to handle shared DHCP including setting nextboot from your DHCP and not using DRP and using DRP as a DHCP Proxy to add nextboot instructions to DHCP requests

I think some of those are in the $faq

FAQs are here > http://provision.readthedocs.io/en/latest/doc/faq-troubleshooting.html

@jpresley I am using DRP on a subnet with existing DHCP server - I just am setting ?next server? and ?default file name? params to appropriate values

@jpresley - proxy dhcp may work for you

has joined #community201803

@lcrozzoli $welcome

Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

:smile:

Good morning @wayneeseguin

Hello and thanks to all. i'm very glad to be here

Hello, all. I've managed to navigate the documentation well enough to create a bundle, upload it, and successfully provision machines. It works beautifully and I appreciate everyone's hard work. I have two questions about the process: 1) I'd like to customize the ubuntu preseed "late_command" but cannot find references to it in the documentation. Nor could I find references to it in the standard collection of parameters (I used "select-kickseed" to customize other parts of the installation which worked perfectly). What is the preferred way to override "late_command"? 2) I have a "discover -> prep-install -> ubuntu-16.04-install -> finish-install -> complete-nowait" workflow to accomplish this. Is this reasonable? Apologies if I should open these questions as Github issues. I know it's always a challenge to document every possible use case. I'd like to help out any way I can so I'd be happy to post these questions as Github issues so they can be organized/triaged.

hi @nkabir - glad to hear you've gotten things rolling !

taking a look at your late_command question

currently - there are two paths for you: 1. with `select-kickseed` - you can define you're own post install script - either replacing the `post-install.sh` call, or adding a second line after it ... 2. you can clone the ubuntu-16.04-install BootEnv, and make the `template` call changes in that - using your cloned BootEnv

However, if you clone the BootEnv, when we make DRP Community Content updates, you won't get those changes for that cloned BootEnv - but the original ubuntu-16.04-install BootEnv will be updated

it looks like we actually call "post-install.sh" twice (erroneously) - we call it once in the BootEnv definition in the `templates` section, then again in the `net-seed.tmpl`

since you are already using `select-kickseed`, the `net-seed.tmpl` that you (presumably) originally cloned can be modified to point to a different `post-install.sh`; and our `post-install.sh` will still run, as it's defined by the BootEnv definition to run last - and you don't want to disturb the `reset-workflow` and `runner` template calls in that, otherwise you'll get bad behavior with the Workflow/Stages and jobs.

Thank you for the clarification, @shane. I'll give that a try!

Also - if you are unsure exactly what a template is going to do - you can render it to see the final product - not everything can be rendered in it's final state, since the context in which it was called matters in some cases

but there is a $faq on test rendering templates

FAQs are here > http://provision.readthedocs.io/en/latest/doc/faq-troubleshooting.html

the rendering works for other things too, any template defined in a BootEnv can be rendered against the "machines" url, example: http://drp.domain.com:8091/machines/817cbf29-30be-4807-b5a0-1234567890/post-install.sh

the trick is ... the machine must be in the Stage that the templates are defined in - so the `ubuntu-16.04-install` Stage - defines templates, put a "phantom" machine in to that bootenv, and then render away

this is true for other stages too - I use a phantom machine to render stuff all the time

I was able to get that far. I wanted to work within DRP preferred conventions and ensure I didn't customize too far afield of the tool's best-practices. Would my custom post-install.sh script need to follow the BootEnv post-install script's configuration of the chroot environment i.e. using a here-doc and executing in /target?

I'd suggest cloning the existing `net-post-install.sh.tmpl`, and work with in the same conventions - just modify in the HereDoc what you want to actually perform - and change the name of the written out file from `update_system2.sh` to something like `update_norms_secret_post_install.sh` - otherwise keeping everything else intact around the HereDoc

:+1: will give that a try!

@nkabir - additionally you could add a new Stage, with a Task `norms-post-install`, which calls the new `post-install` script - you'd then insert that in your work flow between `ubuntu-16.04-install` and `finish-install`

@shane I like your suggestion better. It's more discoverable and self-documenting. Thanks!

The Stage/Task/Template solution?

^^^ was @greg and @vlowther reminding me that is the better path to follow ... :slightly_smiling_face:

@shane Speaking of stages and tasks, I'm having a problem with a custom Stage/Task as follows...

I made a custom stage based on `prep-install` named `necla-prep-install` which is as follows:

@wdennis uploaded a file: https://rackn.slack.com/files/U416T0AAX/F9Q60RWN6/necla-prep-install.txt

This uses the Task I named `totally-erase-sda` which is as follows:

@wdennis uploaded a file: https://rackn.slack.com/files/U416T0AAX/F9R1HV37G/totally-erase-sda.txt

I wired the stage up in a stage-map in my profile as follows:

@wdennis uploaded a file: https://rackn.slack.com/files/U416T0AAX/F9Q56J8J0/stage-map.txt

So, as I understand it, this will boot SH, then execute the template script in the task called by the `necla-prep-install` stage, which should do the `dd` command which will zero out /dev/sda. After this completes, the system will reboot, and go into the Ubuntu 16.04 install.

The systems calling the profile with the stage map are booting SH, but then nothing is executing (no `dd` is happening.) Why might this be?

I should also mention that `necla-prep-install` is a modified clone of `prep-install`, and `totally-erase-sda` is a modified clone of `erase-hard-disks-for-os-install`

The prior stage map was calling the `prep-install` Stage, which was successfully executing the template in `erase-hard-disks-for-os-install`, but the disk wipe was not sufficient, and the Ubuntu install thereafter was failing.

I have found that if I do a `dd if=/dev/zero of=/dev/sda bs=512` then all goes smoothly with the resulting install

not sure off the top of my head, I'm out at dinner right now - will look at this a little bit later

Mornin', DR folk!

Any ideas on my problem above? ^^^

sorry - had client work I was dealing with didn't get a chance to look at it ... will try and take a peek soon

@shane Thx

OK, I give up... Pretty sure it's not a DRP problem (except for the disk wiping issue not working as above) - dd'd the disks with zeros manually and the Ubuntu preseed still crapping out. Going to try MAAS and see if I have a different experience.

I believe Ubuntu writes the partition tables in a way that is hard to undo. Greg has had to fix it for other people. I don't know the details.

@zehicle Zeroing the disk should overwrite all of that, yes? I don't believe I ever ran into this problem though using Cobbler to reinstall previously-installed machines, and of course doing a manual install via USB on a previously-used disk work fine...

It's just that I have machines to reinstall, and I can't get DRP to work reliably on used disks...

There is data written in places you can't easily find. Sorry I don't have the details. It's a Ubuntu install issue.

I'm not assuming that I google better than you -> there's something about how the partition table is written.

It took Greg a while to fix it before (I remember him describing it as "blah Ubuntu blah Drive Paritions blah install turds blah" so I know it's a thing.

Greg never says turds, that's my color commentary

@wdennis I am running Ubuntu 16.04 installs (and re-installs) on a set of machines to familiarize myself with the tool. I started with "discover (start) -> prep-install (reboot) -> ubuntu-16.04-install (reboot) -> finish-install (reboot) -> complete-nowait (success)" before venturing into custom stages. To restart the process, I'm executing "# dd if=/dev/zero of=/dev/sda bs=1024 count=1", resetting the DRP Machine entry stage to "discover", and rebooting. It appears to re-install successfully. Have you managed to get a stock life-cycle working? I've noticed it's easy for me to make typos in "shell + template" files. I started out with MAAS until I discovered DRP. I prefer DRP.

@nkabir Yes, your workflow was just about exactly like mine -- but I find that `prep-install` doesn't do enough to clear the disk for re-use (I'm using LVM, and I'm pretty sure that's causing the issue.) I have done "stock" DRP-provided template installs, and they do work. But, I cannot stick with that due to my partitioning needs (as well as other preseed needed customizations.) I realize that RackN doesn't support debugging preseed problems, but I'm loosing too much time on this trying to figure it out myself... So I'm going to do a test using MAAS and see what my experience is there. Maybe I'll quickly come back to DRP :wink:

`prep-install` seems to work for many. It would be good to see your workflow again @wdennis. I?m out though and will get yelled at for this message.

@greg - go back to your vacation

Wow, that didn't take long...

@nkabir - I think you can remove the `prep-install(reboot)` with `prep-install(success)` - one less reboot that way. `finish-install(reboot)` should be `finish-install(stop)`, but with the tip code base it will ?fix? it for you.

hi guys i'm back. I had dr-provision working well with mellanox cards, but now on a machine with SolarFlare, I'm getting stuck.

I have the option 67 override, which I needed for mellanox: `67: "{{if (eq (index . 77) \"iPXE\") }}default.ipxe{{else if (eq (index . 93) \"0\")}}ipxe.pxe{{else}}ipxe.efi{{end}}"`

I have only serial console to the machine I'm trying to provision. the screen goes blank.

i have a tcpdump on the drp box, and it shows tftp transfer of ipxe.pxe, then nothing

Ah I got someone to take a pic of the screen. I have this, and it froze ```PXE->EB: !PXE at (AC0:0790, entry point at 9AC0:0248 UNDI code segment 9AC0:0838, data segment 9B44:1c?? UNDI device is PCI 06:00.0, type gPXE 619kB free base memory after PXE unload Oops! Unable to find realmode segment```

@greg Thank you. That will save me some time as I iterate through my set up! Now back to enjoying Mai Tais (or equivalent)! @wdennis I am using LVM on a single disk but I haven't ventured into customizing partitioning yet. My staging machines have a single boot disk (`/dev/sda`) and three additional disks that are configured for ZFS/RAIDZ by Ansible once the DRP provisioning sets up the single boot disk. Ansible handles the partitioning of ZFS. I just need the boot disk to host the OS.

hm removing option 67 worked for this box.

so hmm. lpxelinux.cfg freezes my mellanox box and ipxe.pxe freezes my solarflare box

that puts a skunk in the works

gotta love the PXE "_standards_" ...

do you have any tricks up your sleeve?

@greg or @vlowther might have some iPXE script options to help - unfortunately, I'm not sure how to implement that right to work for both cards

What nic is in that solarflare box?

solarflare is the nic

...

Interesting.

sorry, having trouble with copy/paste

Solarflare SFC9020

I see option 67 has template interpolation -- would we be able to switch on gohai inventory?

I assume they work fine once we get booted into Sledgehammer?

although that doesn't help for discovery

yup

Alas, the template interpolation that happens for option 67 only has access to the incoming DHCP packet.

hm.

What firmware are those solarflare nics running?

soo, we were pxe booting both types with our own pxelinux.0 via tftp

maybe I can tweak option 67 to have it try pxelinux.0 instead of lpxelinux.0 / ipxe.pxe

Trolling through the firmware release notes indicate that they have had a few firmware updatres to fix PXE related issues.

trying to dig up the version

release notes for their latrest utility bundle.

It looks like they have use an embedded gPXE for their PXE booting needs.

i have this so far, still trying

```Solarstorm Boot Manager (v3.2.0.6061) Solarflare Communications 2008-2010 ```

reading those release notes

heh, later releases allow you to embed your own custom ipxe ROM image.

COuld have all sorts of fun with that. :slightly_smiling_face:

ah trying to avoid that kind of fun for now :slightly_smiling_face:

``` Firmware version: v3.2.1 Controller type: Solarflare SFC9000 family Controller version: v3.2.0.6071 Boot ROM version: v3.2.0.6061 ```

quite far behind

In the mean time, though, if you have a version of pxelinux and/or ipxe that work for both systems in question you could just chuck them into /var/lib/dr-provision/tftpboot and rewrite your option 67 to use them instead.

ok thanks

or...

sorry, bad advice.

Make /var/lib/dr-provision/replace. and name then the same as their respective files in /var/lib/dr-provision/tftpboot

sorry, I am in the bad habit of just recompiling when I want to test different embedded assets

so mkdir /var/lib/dr-provision/replace; cp pxelinux.0 /var/lib/dr-provision/replace/mypxelinux.0, then change option 67 to send mypxelinux.0

no, I think it will have to be named lpxelinux.0

and you will need to include the related .c32 files for pxelinux

ipxe is easier because it will not try to load binary modules.

but I bet you will need a firmware update to get ipxe working.

oooh i c. so /var/lib/dr-provision/replace masks /var/lib/dr-provision/tftpboot?

i'm bricking, er, working on upgrading the firmware

That is the idea, yes. :slightly_smiling_face:

does sledgehammer on serial console have password login?

root/rebar1

silly me did the firmware update over ssh and of course lost connection

omg thank you

YES! firmware update seems to have done it

@vlowther thank you for heroic advice :slightly_smiling_face:

:slightly_smiling_face:

And it looks like the latest firmware has a slew of fixes and enhancements over your older version.

ok i'll come back again when I get stuck

:slightly_smiling_face:

@wdennis Is it still the --force --force --I-really-mean-it thing?

@nkabir our erase script is a little more paranoid than zeroing the first megabyte: https://github.com/digitalrebar/provision-content/blob/master/content/tasks/erase-hard-disks-for-os-install.yaml

we iterate over all the vgs, forcibly erase them, do the same for all the pvs

then wipe out the first and last meg of every partition and raw block device.

wat?

You had a similar issue a few weeks ago where vgremove was failing because one --force was not enough.

So my solution was to zeroize the entire disk via `dd if=/dev/zero of=/dev/sda bs=512`

That should do the trick, right?

Yes, the main reason I don't want to do that by default is because it can take hours if (for example) you are zeroing a multi-terabyte drive.

so I would prefer an approach that is a bit more targeted in what it erases

Except that it doesn't... The Ubuntu install I am trying to do afterwords still craps out...

possibly with fixes to the seed files for Debianoids that tell the install to really ignore anything that might already be on the disk.

Yes, I'd prefer a shorter wipe as well, but only if it works...

but you know all about the lack of documentation there. :confused:

I was using your `prep-install` stage but was experiencing faults when the installer got to the partitioning step

The interesting thing is that a standard USB install (i.e. a regular ISO installer) doesn't complain about anything when you use a used disk

It just partitions however you told it to and moves on

So it probably knows about seed options we don't

yes

I need a specific partitioning to support how we want to configure the boot/root disk

It's basically:

``` d-i partman-auto/expert_recipe string root_home_lvm : \ 500 533 1024 free \ $iflabel{ gpt } $reusemethod{ } \ method{ efi } format{ } \ . \ 1024 1024 1024 ext2 \ $primary{ } method{ format } \ format{ } use_filesystem{ } filesystem{ ext2 } \ mountpoint{ /boot } \ . \ 1 1073741824 -1 ext3 \ $defaultignore{ } method{ lvm } \ vg_name{ vg00 } \ . \ 50% 20 100% linux-swap \ $lvmok{ } \ lv_name{ lv_swap } in_vg{ vg00 } \ method{ swap } format{ } \ . \ 204800 204800 204800 ext4 \ $lvmok{ } in_vg{ vg00 } \ lv_name{ lv_root } method{ format } \ format{ } use_filesystem{ } filesystem{ ext4 } \ mountpoint{ / } \ . \ 512000 512000 512000 ext4 \ $lvmok{ } in_vg{ vg00 } \ lv_name{ lv_home } method{ format } \ format{ } use_filesystem{ } filesystem{ ext4 } \ mountpoint{ /home } \ . ```

So it makes LV's for "/", swap and "/home"

The idea is to make it partition any size of disk (we have very hetero hardware here...)

has joined #community201803

Basically make a reasonable-sized "/" LV, reasonable-sized swap LV, then all the rest of the VG space goes to the "/home" LV

yes

it is finding the right partman-lvm and partman-md and other related options to make it really clear everything and not ask questions

We need that because the researchers get local home dir's on the servers, and tend to pile lots of data in them... If "/home" was a part of the "/" LV, they run the OS partition out of space and crash the servers...

It's *so* easy in feekin' kickstart... Preseed is so obtuse.

But, the reseachers all want Ubuntu OS...

has joined #community201803

@tim_epkes $welcome

Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

Thanks

has joined #community201803

@vlowther Yes. It's very thorough! I'm repeatedly cycling through the whole process in my staging environment to get more comfortable with the moving parts. I run your erase stage in my workflow. However, I only erase the first megabyte when I want a machine to return to discovery via PXE. Then the workflow takes over.

@nkabir uploaded a file: https://rackn.slack.com/files/U8BTZ6HPT/F9RAUAV4J/-.sh and commented: @wdennis with your successful manual install, have you dumped the working machine's configuration and compared it with your DRP template?

@fpisupati $welcome

Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

thanks

@nkabir commented on @nkabir?s file https://rackn.slack.com/files/U8BTZ6HPT/F9RAUAV4J/-.sh: @wdennis You may also want to try wipefs https://www.cyberciti.biz/faq/howto-use-wipefs-to-wipe-a-signature-from-disk-on-linux/

Random/unrelated : Was browsing your website and found a nice pict of you @shane @zehicle @greg https://www.rackn.com/company/ ^^ I felt it was worth to share with the community

LOL ... yeah, well @zehicle had a "wardrobe malfunction" and didn't end up wearing his kilt that day ... that's @vlowther in the middle, @greg on the right, and /me on the left ...

^^

These are unapproved marketing show uniforms :grinning:

s/unapproved/maverick/g

The strange part is that the merely average guy is short.

And comfortable

has joined #community201803

@patrick.miller $welcome

Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

thanks zehicle

indeed, welcome, @patrick.miller :slightly_smiling_face:

how can I display the kickstart file for a machine?

the rendered one - or the template that builds the final KS ?

rendered

excellent thanks!

there are a lot of nuggets in the $faq

FAQs are here > http://provision.readthedocs.io/en/latest/doc/faq-troubleshooting.html

(oops, I need to update Slackbot responses to point to `tip` doc instead of `latest`)

ok - done: $quickstart

QuickStart doc is here > http://provision.readthedocs.io/en/tip/doc/quickstart.html

@patrick.miller in a short bit, the `tip` docs will update with some clean ups and enhancements to that Render doc - but the info is correct as you see it right now

@wdennis - I got a chance to look at your Erase SDA issue ... the problem is you set `sane-exit-codes` - we expect appropriate Exit Codes to mark success. However - our "insane" (aka older Exit Code definitions) also assume Exit Code of 0 (zero) for success. The `dd` command will always exit with code ` 1 ` and message "out of disk space". This causes the Agent/Runner to mark the job as failed, and your Workflow won't advance. By explicitly setting an `exit 0` at the end of the scriptlet will fix your issue. For reference, I am attaching a content pack that contains the testing I did. NOTE - I moved your scriptlet out of the payload of the Task, and in to a separate Template, as it makes it easier to extend/edit/track/modify, etc. But the principle should be the same (I did not test using the scriptlet embedded in the task w/ a modified `exit 0`). I'm including 2 workflows - your original NECLA and a modified `rebar-prep` workflow that does the same thing, but uses the Digital Rebar/RackN `erase-hard-disks-for-os-install` Task. If this needs modification to work right - we'd appreciate the feedback on what needs to change. It's 1000x's of times faster than a pure-`dd` solution to bigger disks.

@shane uploaded a file: https://rackn.slack.com/files/U6QFVRJNB/F9REST4BE/NECLA_and_Rebar_wipe_prep_content_.yaml and commented: Use like regular content pack. First delete your conflicting content with the same names. Then: `drpcli contents create -< necla-and-rebar-prep.yaml` Attach the appropriate Workflow stagemap to your machines, and boot as usual.

@wdennis commented on @shane?s file https://rackn.slack.com/files/U6QFVRJNB/F9REST4BE/NECLA_and_Rebar_wipe_prep_content_.yaml: OK, thx @shane

has joined #community201803

@scsikid $welcome !

Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

ah ok thanks Shane.

thanks @shane