ctrees
2018-04-02 16:49
Hey all... so I've got 2 requests: 1 - GUI testing (time contribution) 2 - Hindsight logging (thinking community package ?)

ctrees
2018-04-02 16:53
I started the GUI testing thing several months ago, and about to drop into more testing for my day-job... If ya'll think it may be useful in the CI chain I can do that.

ctrees
2018-04-02 16:54

ctrees
2018-04-02 16:57
After talking with Trink, we decided I should do a syslog native aggregation and make sure it can be utilized by provision

ctrees
2018-04-02 17:00

dave.parker
2018-04-02 21:32
I haven't tried without `set filename` I don't think. I'll try that and see.

dave.parker
2018-04-02 21:33
In the meantime, I'm now having a new error trying to install on a Dell R430 using the Ubuntu-16.04-install bootenv. I get errors at the start of the install about /dev/sda not being able to be mounted. "no media found".

2018-04-02 21:33
Time to feed the :bear:!

dave.parker
2018-04-02 21:34
The install then fails at partitioning

dave.parker
2018-04-02 21:34
But the standard Ubuntu installer finds /dev/sda just fine.

shane
2018-04-02 21:34
in the installer stage, are you sure Ubuntu iterates as SDA? maybe only after reboot does it show as SDA ?

dave.parker
2018-04-02 21:35
Not sure what you mean.

shane
2018-04-02 21:37
neither do I ...

shane
2018-04-02 21:38
:slightly_smiling_face: what I meant was ... during the installar boot/install process, sometimes devices are identified differently than when they boot in to the installed OS instance ... but if you see it as /dev/sda *during* the Ubuntu installer (in the working example) ... than that's likely not the case

shane
2018-04-02 21:38
we are using Ubuntu's own ISO media image to PXE boot the machine - so they should be equivalent to what you are doing (in the working scenario)

dave.parker
2018-04-02 21:40
Ok. Well let me boot the vanilla installer again and double check the drive names.

shane
2018-04-02 21:52
what version of Ubuntu are you using (that works) ?

shane
2018-04-02 21:53
if you are using the results of the `drpcli bootenvs isoupload ubuntu-16.04-install` command (or equivalent) to install from DRP - that is the official Ubuntu 16.04.4 ISO image

shane
2018-04-02 21:53
we don't have a "customized" version of Ubuntu we try and install

shane
2018-04-02 21:54
it's possible our Boot Flags are slightly different which might be not unlocking some of the Installer options

shane
2018-04-02 21:54
can you capture the boot flags to the kernel on the default installer ?

dave.parker
2018-04-02 22:26
Ok. Still trying to verify the vanilla installer sees the disks as sda etc. It's slow going.

shane
2018-04-02 22:26
indeed ... :disappointed:

shane
2018-04-02 22:27
I don't recall if the Ubuntu installer will capture the boot flags somewhere - if so that may help - because the only real difference is going to be the kernel options/flags we pass - which are defined in the BootEnv

dave.parker
2018-04-02 22:27
Building a server several thousand miles away with an ISO over the internet is pretty much exactly why I want to get dr-prov working.

shane
2018-04-02 22:28
have you overridden the install device in the past ? we do parameterize the Preseed to let you override the install device ...

shane
2018-04-02 22:28
just want to make sure there isn't any left over cruft from a previous install that might have specified a different install device ?

dave.parker
2018-04-02 22:28
Yes I've had to do it for kvm guests.

shane
2018-04-02 22:28
for example Xen VMs require different devs

shane
2018-04-02 22:28
likely setting to /dev/vda ?

dave.parker
2018-04-02 22:28
Yup

shane
2018-04-02 22:29
hopefully you didn't set that on the Global Profile ?

dave.parker
2018-04-02 22:29
No

dave.parker
2018-04-02 22:31
So when I booted off the vanilla installer, I got those same errors about being unable to mount /dev/sda and such. But the partitioner in the installer definitely sees the disks as sda sdb sdc and sdd

dave.parker
2018-04-02 22:31
Let me see if I can do partitioning. There are partitions on here as if the partitioning part of the install worked.

shane
2018-04-02 22:31
if you have a real (or fake) machine set to the ubuntu-16.04-install BootEnv, you can render the Preseed that would be served to it - see: http://provision.readthedocs.io/en/tip/doc/faq-troubleshooting.html#render-a-kickstart-or-preseed

dave.parker
2018-04-02 22:33
Ok

dave.parker
2018-04-02 22:34
Well partitioning worked with the vanilla installer.

dave.parker
2018-04-02 22:40
The preseed is definitely trying to use /dev/sda

dave.parker
2018-04-02 22:42
I've tried the default net-preseed and a custom one I built specifically for this type of server, to set up software raid.

dave.parker
2018-04-02 22:42
It really looks like the partitioning works.

dave.parker
2018-04-02 22:42
Because there are partitions everywhere on these disks now. :slightly_smiling_face:

dave.parker
2018-04-02 22:42
So I'm not sure what's failing now. I guess let me try another install and see what exactly the error is. I know it was something about being unable to mount the root filesystem.

dave.parker
2018-04-02 22:50
Hrm. The error is "No root file system is defined"

shane
2018-04-02 22:50
this might be related to partitioning scheme in the Preseed

dave.parker
2018-04-02 22:51

dave.parker
2018-04-02 22:51
That's the partitioning section of the rendered preseed

shane
2018-04-02 22:51
the one that fails for you, right ?

dave.parker
2018-04-02 22:52
Yes.

dave.parker
2018-04-02 22:52
That's the one I just tried.

shane
2018-04-02 22:52
are you really running IPv6 in that environment ?

shane
2018-04-02 22:52
(line 35)

dave.parker
2018-04-02 22:52
Huh.

dave.parker
2018-04-02 22:53
Yes, but I'm not sure it's getting a v6 IP from DHCP...

dave.parker
2018-04-02 22:53
Why is it defaulting to that?

dave.parker
2018-04-02 22:54
And the dr-prov machine does not have a v6 address defined.

dave.parker
2018-04-02 22:54
So that seems unlikely to work.

shane
2018-04-02 22:54
'tis odd indeed

shane
2018-04-02 22:55
it'd be failing to pull down the filesystem.squqshfs with that

shane
2018-04-02 22:55
do you have Routers broadcasting IPv6 on that segment ?

dave.parker
2018-04-02 22:56
I would say that's likely. But I'm not sure honestly.

dave.parker
2018-04-02 22:56
Can I force it to use v4 somehow?

shane
2018-04-02 22:56
well - you can force your underlying Linux OS to be v4 only - varies from distro-to-distro a little bit

dave.parker
2018-04-02 22:56
Looks like that's set from InstallURL

shane
2018-04-02 22:57
within DRP - we query your NICs to pick up clues on how to handle replying back to Machines

shane
2018-04-02 22:58
we dynamically build tables on how to return to the Machine based on port info - and that propagates up in to the Templates

dave.parker
2018-04-02 23:01
So there's no way to override that?

shane
2018-04-02 23:02
Cloning your net-seed.tmpl and customizing it to not dynamically inject the DRP endpoint address in that template field ...

shane
2018-04-02 23:02
Disabling IPv6 completely in the Base OS that DRP is running on should also do it

shane
2018-04-02 23:02
you'd probably need to restart DRP after disabling IPv6 on the DRP endpoint host

dave.parker
2018-04-02 23:02
Ok.

dave.parker
2018-04-02 23:03
I'll try just changing the template first.

shane
2018-04-02 23:03
we'd need @vlowther to weigh in on any other secret sauce that I'm not aware of ....

shane
2018-04-02 23:05
but in the end - we pick up that address from the NICs and building our own internal tables of what address to set the `{{.Env.InstallUrl}}` Parameter to

shane
2018-04-02 23:05
that's line 123 in the stable (v3.7.3) Ubuntu `net-seed.tmpl`

vlowther
2018-04-02 23:06
Huh, the only scenario that we would hand back an ipv6 address is if that system talked to us via ipv6 first.

dave.parker
2018-04-02 23:17
I get that `[::1}` just doing a curl from the dr-prov machine using localhost:8091

amr
2018-04-02 23:17
has joined #community201804

shane
2018-04-02 23:18
@amr $welcome :slightly_smiling_face:

2018-04-02 23:18
Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

amr
2018-04-02 23:18
Thanks Shane

shane
2018-04-02 23:18
@dave.parker yep - which is what led me down the ipv6 route .. :slightly_smiling_face:

scsikid
2018-04-02 23:18
@dave.parker i didn?t read everything, did you get your sda issue sorted?

dave.parker
2018-04-02 23:18
Sort of.

dave.parker
2018-04-02 23:18
Sorting that out has lead us down this ipv6 path.

scsikid
2018-04-02 23:19
whats wrong with v6? no worky?

dave.parker
2018-04-02 23:19
I'm just going to hardcode the URL in the template and see if that works.

shane
2018-04-02 23:19
Summary is - DRP believes we need to reply back to the Machine being provisioned via IPv6 "localhost" route - which is wrong

shane
2018-04-02 23:19
it should fix that specific failure

shane
2018-04-02 23:20
but we still have the IPv6 mystery we'd like to sort out

dave.parker
2018-04-02 23:20
I added in a v6 address for the DRP server and rebooted/restarted DRP and it still wants to give out that bogus URL.

scsikid
2018-04-02 23:23
wanna try disabling v6?

shane
2018-04-02 23:24
'twas my suggestion ...

scsikid
2018-04-02 23:24
if so i?ve changed the following in /etc/sysctl.conf ```net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1```

scsikid
2018-04-02 23:24
i think there is another way which is by modifying the /etc/gai.conf file so it asks for an A record instead of a AAAA record.

scsikid
2018-04-02 23:25
i haven?t done that, but i?d suspect it would work

scsikid
2018-04-02 23:25
my guess is the dns resolver you?re using returns both A and AAAA records, and the network stack prefers AAAA in ubuntu.

scsikid
2018-04-02 23:25
you can change that though :wink:

dave.parker
2018-04-02 23:26
We can try that.

scsikid
2018-04-02 23:26
:thumbsup:

dave.parker
2018-04-02 23:26
Editing the template is a pain.

scsikid
2018-04-02 23:26
yea i?d assume you?ll get V6 working one day.

shane
2018-04-02 23:26
I don't think the /etc/gai.conf will work

scsikid
2018-04-02 23:26
no?

shane
2018-04-02 23:27
we parse the packets we get and build our own tables to determine how to reply back to the Machine being provisioned

scsikid
2018-04-02 23:27
but can?t the client request just an A record?

scsikid
2018-04-02 23:27
saying? I only want to know the A record not ANY record

shane
2018-04-02 23:27
we dynamically build the various Template pieces we need based on this info on the fly as the Template is served when requested

shane
2018-04-02 23:28
the client request gets the traffic *from* the Machine (client) to the DRP Endpoint (server) ... but on the DRP Endpoint side we do our own processing to determine _which_ information (IP address) to serve inside the Templates

scsikid
2018-04-02 23:28
ah

scsikid
2018-04-02 23:28
ok

shane
2018-04-02 23:29
because we deal with many complex network scenarios where the DRP Endpoint is multi-homed and we need to calculate how to build the Templates for any given inbound request from different places

dave.parker
2018-04-02 23:31
Now I get this: `d-i live-installer/net-image string http://127.0.0.1:8091/ubuntu-16.04/install/install/filesystem.squashfs`

dave.parker
2018-04-02 23:31
Which doesn't seem like it's gonna work either. Why is it using localhost?

shane
2018-04-02 23:31
DRP is calculating that based on the client connection - but I agree with you, that sounds very wrong

shane
2018-04-02 23:32
ironically - that does correlate with the IPv6 address we calculated

dave.parker
2018-04-02 23:32
Yeah, sure does.

shane
2018-04-02 23:32
what version of DRP are you running? (`drpcli info get | jq '.Version'`)

shane
2018-04-02 23:32
(or Info & Prefs in Portal)

dave.parker
2018-04-02 23:32
I get... null

dave.parker
2018-04-02 23:33
That seems wrong too!

dave.parker
2018-04-02 23:33
Versions: DR v3.7.3 & UX v1.0.2

dave.parker
2018-04-02 23:33
Is what I see in the gui

shane
2018-04-02 23:33
sorry - lowercase 'v' in version

dave.parker
2018-04-02 23:34
`"v3.7.3-0-5b1774b63a9abce5c7ae43d5dbd901b7b1439c68"`

shane
2018-04-02 23:34
can you pull your `ps` listing for `dr-provision` on that endpoint ?

dave.parker
2018-04-02 23:34
```$ ps -ef | grep dr-provision root 1030 941 0 23:29 pts/3 00:00:00 sudo ./dr-provision --base-root=/home/ubuntu/dr-prov/drp-data --local-content= --default-content= root 1031 1030 0 23:29 pts/3 00:00:01 ./dr-provision --base-root=/home/ubuntu/dr-prov/drp-data --local-content= --default-content=```

shane
2018-04-02 23:34
we need to verify if the `--base-root=...` is pointing to the correct location

dave.parker
2018-04-02 23:35
Looks right to me.

shane
2018-04-02 23:35
is `/home/ubuntu/dr-prov/drp-data` the correct installed location ?

dave.parker
2018-04-02 23:35
```$ pwd; ls /home/ubuntu/dr-prov/drp-data digitalrebar job-logs plugins replace saas-content tftpboot ux```

dave.parker
2018-04-02 23:36
I'm gonna get some dinner, be back in a few.

shane
2018-04-02 23:36
when was the last time you restarted DRP ?

shane
2018-04-02 23:38
can you please run: ` drpcli interfaces list`

shane
2018-04-02 23:38
(hopefully before you bounce the service)

dave.parker
2018-04-02 23:41
```$ ./drpcli interfaces list [ { "ActiveAddress": "100.84.10.7/26", "Addresses": [ "100.84.10.7/26" ], "DnsDomain": "http://dyndns.com", "DnsServers": [ "100.111.250.10", "100.111.252.10" ], "Gateway": "100.84.10.1", "Index": 3, "Meta": null, "Name": "v2000-0-0044", "ReadOnly": true } ]```

dave.parker
2018-04-02 23:42
Ok, I got food and I'm back.

shane
2018-04-02 23:43
ok - talk in a bit

dave.parker
2018-04-02 23:43
I restarted drp just recently.

dave.parker
2018-04-02 23:43
When I rebooted to disable ipv6

dave.parker
2018-04-02 23:44
I'm going to work on hardcoding the info into the template just so I have a workaround for this.

shane
2018-04-02 23:44
You should be able to just clone the template, change that one Template reference

dave.parker
2018-04-02 23:44
I'll continue to help troubleshoot the base mystery though since i'm curious and don't like the hack of the clone.

shane
2018-04-02 23:44
but you also have to clone Stage -> BootEnv -> Template - be able to point at a new template

dave.parker
2018-04-02 23:46
That `{{.Env.InstallUrl}}` is referenced a bunch of times in the template

shane
2018-04-02 23:46
yes - each of those references

vlowther
2018-04-02 23:49
Sorry, making dinner. @ @dave.parker if localhost resolves to an ipv6 address we will talk back on ipv6, and generate ipv6 urls.

vlowther
2018-04-02 23:50
What matters is if the hosts are talking to us over ipv6, and if your environment handles that properly.

vlowther
2018-04-02 23:54
We do not use dns at all to figure that out, we look at the remote address and interface info the kernel gives us for the socket.

scsikid
2018-04-02 23:55
:thumbsup:

dave.parker
2018-04-02 23:56
So should I try to rediscover the machine or something?

vlowther
2018-04-02 23:57
Are we handling DHCP?

vlowther
2018-04-02 23:58
Sorry, I don't know enough about the specific environment.

vlowther
2018-04-03 00:04
In unrelated news, I just cut a pack of hot dogs open with a hatchet.

scsikid
2018-04-03 00:04
skill ^

vlowther
2018-04-03 00:05
It is grilling weather in Austin.

dave.parker
2018-04-03 00:05
No, DHCP is being handled by another server.

vlowther
2018-04-03 00:07
Ok. If you crank general and render logging up to debug the drp logs will have everything that gets rendered, and the log will also contain address cache entries as they are created and used.

scsikid
2018-04-03 00:08
i now want a hot dog. and austin weather.

2018-04-03 00:08
Who let the dogs out? :dog: :dog: :dog:

vlowther
2018-04-03 00:08
The steaks are for the adults.

dave.parker
2018-04-03 00:09
No thanks on the Austin weather. I hear it's miserable in the summer.

vlowther
2018-04-03 00:09
My kids don't like fresh seared cow for some reason.

scsikid
2018-04-03 00:09
i eat all the things, and i love heat.

dave.parker
2018-04-03 00:09
Do you love humidity?

scsikid
2018-04-03 00:09
i?m ok with it.

scsikid
2018-04-03 00:09
prefer it then the cold.

dave.parker
2018-04-03 00:10
How do I set the logging level?

dave.parker
2018-04-03 00:10
Im trying the edited template right now.

dave.parker
2018-04-03 00:11
Well, the edited template didn't work either.

dave.parker
2018-04-03 00:11
Well...

dave.parker
2018-04-03 00:12
That's because something's wrong, since the wrong template is still being used. So I gotta fix that...

vlowther
2018-04-03 00:16
I the us look under info and preferences.

vlowther
2018-04-03 00:17
In the cli, use drpcli prefs.

shane
2018-04-03 00:19

shane
2018-04-03 00:20
basically - just set a Param on the Machine (or a profile attached to the Machine) named `select-kickseed` to the name of the cloned template (net-seed.tmpl)

dave.parker
2018-04-03 00:20
Yeah, I set the select-kickseed on the machine to my template, but it didn't work. Then I tried cutting out select-kickseed entirely and set the presseed in my custom bootenv to my template. But I still get the wrong info when I do the render check.

vlowther
2018-04-03 00:21
What precisely is wrong?

shane
2018-04-03 00:21
still get 127.0.0.1 for `{{.Env.InstallUrl}}`

vlowther
2018-04-03 00:22
From somewhere besides the machine dr-provision is running?

dave.parker
2018-04-03 00:22
Yeah. I cut out .Env.InstallUrl entirely from my custom preseed template and just hard coded the URL.

shane
2018-04-03 00:22
@dave.parker - "where" is your Machine located relative to DRP ?

dave.parker
2018-04-03 00:23
But I still see 127.0.0.1 in the rendered information when I query for it.

vlowther
2018-04-03 00:23
Getting a loopback IP is expected when querying it from localhost.

dave.parker
2018-04-03 00:23
Oh ok.

dave.parker
2018-04-03 00:24
Well no, it shouldn't show up at all now.

dave.parker
2018-04-03 00:24
Because I'm not using that variable, I'm hard coding it.

vlowther
2018-04-03 00:24
We generate that IP based on the other side of that specific tcp connection.

vlowther
2018-04-03 00:25
And the other side of localhost is llocalhost.

dave.parker
2018-04-03 00:26
The line in the default preseed is `d-i live-installer/net-image string {{.Env.InstallUrl}}/install/filesystem.squashfs` I changed that to `d-i live-installer/net-image string http://100.84.10.7:8091/install/filesystem.squashfs` in my template

dave.parker
2018-04-03 00:27
So shouldn't I get that second URL no matter what now when I do `curl http://localhost:8091/machines/e5243261-ed78-4c2a-a841-5f7340da7bd9/seed`

vlowther
2018-04-03 00:27
What I am saying is that you are probably chasing a wild goose.

dave.parker
2018-04-03 00:28
Yeah... I mean that could be the case.

dave.parker
2018-04-03 00:28
So, let me back up.

dave.parker
2018-04-03 00:28
I'm going to change everything back to default.

dave.parker
2018-04-03 00:28
And try to get more verbose logs.

dave.parker
2018-04-03 00:28
Maybe that will tell me something.

dave.parker
2018-04-03 00:34
Do I have to restart after I change the log levels?

vlowther
2018-04-03 00:34
See

vlowther
2018-04-03 00:34
Probably best to. Less clutter that way.

dave.parker
2018-04-03 00:34
Ok

dave.parker
2018-04-03 00:39
Well that did nothing. No info at all in the logs.

vlowther
2018-04-03 00:40
Ok. Have you also rebooted wahtever system is having the problem?

dave.parker
2018-04-03 00:41
Yes.

dave.parker
2018-04-03 00:41
I'm going to just set everything to debug.

vlowther
2018-04-03 00:44
Fair enough.

shane
2018-04-03 00:45
fair warning ... don't leave everything at debug for too long ... you will grind your DRP instance to a halt eventually

dave.parker
2018-04-03 00:45
lol ok

shane
2018-04-03 00:45
`trace` level lets that happen even faster :slightly_smiling_face:

dave.parker
2018-04-03 00:49

dave.parker
2018-04-03 00:49
Well, that's all I get.

dave.parker
2018-04-03 00:49
Plus side is, the URL is correct...

dave.parker
2018-04-03 00:50
So unfortunately that was a wild goose chase.

vlowther
2018-04-03 00:51
That looks saneish at first glance.

dave.parker
2018-04-03 00:51
Now I'm back to trying to figure out why I get "No root file system is defined. Please correct this from the partitioning menu."

vlowther
2018-04-03 00:51
Ah. Partitioning in preseeds.

vlowther
2018-04-03 00:52
Many people hate it with a burning passion.

dave.parker
2018-04-03 00:52
If I walk through a vanilla Ubuntu 16.04 install I can select the default "whole disk plus lvm" method and it works fine.

vlowther
2018-04-03 00:52
I am one of those.

dave.parker
2018-04-03 00:53
Yeah I'm not super fond of it either. It never seems to work right. Unfortunately I have to do it because we have our special snowflake way of setting up the root disks. But this is the default net-seed and it's not working either. :disappointed:

vlowther
2018-04-03 00:53
Is sda correct?

dave.parker
2018-04-03 00:54
Yes, as far as I know. That's how it shows up in the vanilla installer anyway.

vlowther
2018-04-03 00:55
Because I could go on about how much it sucks that disknordering in Linux is only usually stable.

dave.parker
2018-04-03 00:55
I know it's doing some partitioning because when I run the vanilla installer and get into the manual partitioning I see the partition scheme there.

vlowther
2018-04-03 00:56
You might want to switch to another vt and look at the install logs.

vlowther
2018-04-03 00:56
Wherever they wind up on Debian installs.

vlowther
2018-04-03 00:57
It has been awhile for me.

shane
2018-04-03 00:57
believe tty4 ... maybe ...

dave.parker
2018-04-03 00:58
Ah ha. Trying to figure out how to do that with the Dell virtual console.

2018-04-03 00:58
Time to feed the :bear:!

vlowther
2018-04-03 00:58
Wish they would do what centos does, and have all that in tmux for a text install.

vlowther
2018-04-03 00:58
So there is working backscroll.

dave.parker
2018-04-03 00:59
Ugh yes.

dave.parker
2018-04-03 00:59
Ok found the logs, and there's no useful scrollback. And all that I'm seeing is it finding a whole bunch of logical volumes and mounting them.

dave.parker
2018-04-03 00:59
But some of them look bogus.

dave.parker
2018-04-03 01:00
I'm going to try booting the discovery image and running the disk nuke thing.

vlowther
2018-04-03 01:00
Yeah, Debian.

vlowther
2018-04-03 01:02
If you find the magic partman config thingies that mean "no really ignore whatever you might find on the disks", I would love a pull request.

dave.parker
2018-04-03 01:02
Yeah that'd be great.

vlowther
2018-04-03 01:02
I have tried quite a few different ones over the years. And none of them seem to catch everything.

dave.parker
2018-04-03 01:03
I know in some of our old cobbler snippets we start off with a brute force "erase everything on every disk" type thing, much like the task in drp

vlowther
2018-04-03 01:03
Sometimes the whole drive dd is the only way.

zehicle
2018-04-03 01:03
I believe there's a stage for that

vlowther
2018-04-03 01:04
And our erase everything task can fail if the Debian. Installer lays down a different disknlayout than the one we erase.

vlowther
2018-04-03 01:05
Because it doesn't recognize that if it had to create a new partitioning label it should just ignore anything else it finds.

vlowther
2018-04-03 01:06
But no, it looks every time.

vlowther
2018-04-03 01:06
sighs.

dave.parker
2018-04-03 01:10
Is there a login that works for the sledgehammer image?

dave.parker
2018-04-03 01:10
This doesn't look super promising: ```Log for Job: 4603e4b5-72f4-406e-86d7-d28a61b65bd9 Starting task erase-hard-disks-for-os-install on e5243261-ed78-4c2a-a841-5f7340da7bd9 Starting command ./erase-hard-disks-for-os-install-erase-disks Command running Error: Error opening /dev/sde: No medium found PARTIAL MODE. Incomplete logical volumes will be processed. Reading volume groups from cache. Found volume group "tester01-vg" using metadata type lvm2 Logical volume "root" successfully removed semid 32768: semop failed for cookie 0xd4d3489: incorrect semaphore state Could not signal waiting process using notification semaphore identified by cookie value 223163529 (0xd4d3489) Logical volume "swap_1" successfully removed Volume group "tester01-vg" successfully removed semid 32768: semop failed for cookie 0xd4d3489: incorrect semaphore state Failed to set a proper state for notification semaphore identified by cookie value 223163529 (0xd4d3489) to initialize waiting for incoming notifications. PV /dev/sda5 lvm2 [744.73 GiB] Total: 1 [744.73 GiB] / in use: 0 [0 ] / in no VG: 1 [744.73 GiB] Labels on physical volume "/dev/sda5" successfully wiped. mdadm: Unrecognised md component device - /dev/sdb3 2048+0 records in 2048+0 records out```

dave.parker
2018-04-03 01:10
I'd like to just get in and nuke it all by hand.

zehicle
2018-04-03 01:10
root/rebar1

dave.parker
2018-04-03 01:12
How long do you think it'll take to write 800GB of zeros?

dave.parker
2018-04-03 01:12
:troll:

vlowther
2018-04-03 01:13
If only.

vlowther
2018-04-03 01:13
I fear multi terabyte spinning rust.

vlowther
2018-04-03 01:14
Kick it off in the morning, come back tomorrow afternoon .

vlowther
2018-04-03 01:15
Not to mention what a horrible idea it is for ssds.

dave.parker
2018-04-03 01:15
Yeah.

dave.parker
2018-04-03 01:16
Well fdisk doesn't show any partitions on these disks anymore.

dave.parker
2018-04-03 01:16
So... maybe?

vlowther
2018-04-03 01:16
Then again, fstrim is usually fine for them.

vlowther
2018-04-03 01:16
One way to find out.

dave.parker
2018-04-03 01:17
Ok let's see how this goes.

dave.parker
2018-04-03 01:18
I'm going to watch the text console now that I've figured out how to get to it.

dave.parker
2018-04-03 01:26
Well, that didn't work.

dave.parker
2018-04-03 01:26
Weirdly the last thing in the log is `partman: Reading all physical volumes. This make take a while...`

dave.parker
2018-04-03 01:27
But it's already errored out.

dave.parker
2018-04-03 01:27
Oh well. I'm done for tonight. I'll pick it up again tomorrow.

dave.parker
2018-04-03 01:27
Thanks for your help everyone.

vlowther
2018-04-03 01:29
Yeah, time to put the kids to bed.

tuxedopenguin
2018-04-03 11:08
has joined #community201804

tuxedopenguin
2018-04-03 12:26
hello. I'm just trying out digital rebar but struggling to find info about two things: - how to make the machines after install to use static IP addresses (preferably for two interfaces, private, public) like with xcat. - and how can I run a command against all the machines aka xcat's xdsh command? thanks

greg
2018-04-03 12:28
The first would be handled by a custom task to set the interfaces up in the OS of your choice. We?ve shied away from writing this because it is often a one-off set of tasks and not general. There are a lot of possible combinations. Doesn?t mean that we shouldn?t have one, just not right now. It could be staged into the process post install or post-install reboot time frames.

greg
2018-04-03 12:29
The second is a philosophical one. DRP doesn?t make remote access a default condition. You can install ssh keys and then use ansible or whatever tool of choice. In general, we?ve tried to enable additional tools without forcing one. @tuxedopenguin

greg
2018-04-03 12:30
WIth that said, after fighting virtualbox, I?m really tempted to make a dhcp all interfaces task so I can have outbound access to the internet from my VMs.

greg
2018-04-03 12:31
The `xdsh` like functionality could be built / scripted around the API, but would require ssh keys to be in place.

tuxedopenguin
2018-04-03 12:31
I'm trying to play with bare meal...

tuxedopenguin
2018-04-03 12:31
I set the ssh keys on the machines

greg
2018-04-03 12:32
The other thing is that the runner can pull tasks, so a maintenance like workflow could also be used for your methodology.

tuxedopenguin
2018-04-03 12:32
post-install would be the perfec one to handle static ips

tuxedopenguin
2018-04-03 12:33
first I tried to create a template, then a task to call that template

tuxedopenguin
2018-04-03 12:33
but then I was not able to run somehow the task

zehicle
2018-04-03 12:56
@tuxedopenguin there's a training video about debug in the library. I'll send the link shortly

tuxedopenguin
2018-04-03 12:56
thanks

zehicle
2018-04-03 13:11
@tuxedopenguin instead of the links directly to the video... here's the github w/ the sample content AND links to the video in the readme https://github.com/digitalrebar/colordemo

tuxedopenguin
2018-04-03 13:36
thanks a lot!

jeff.holland
2018-04-03 16:04
has joined #community201804

zehicle
2018-04-03 19:20
PSA: If you are playing w/ Workflows in tip, the http://test.rackn.io UX now has support the the workflow features.

shane
2018-04-03 19:20
@jeff.holland $welcome

shane
2018-04-03 19:20
$welcome :slightly_smiling_face:

2018-04-03 19:20
Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

shane
2018-04-03 23:11
@wdennis...hopefully this will make you happier... https://github.com/digitalrebar/provision-content/pull/74

dave.parker
2018-04-04 00:29
Hey folks.

dave.parker
2018-04-04 00:29
Update on the issue I was troubleshooting last night.

dave.parker
2018-04-04 00:30
Turns out, something on the box shows up as a USB drive, and that gets set up as /dev/sda

dave.parker
2018-04-04 00:30
So that's why nothing would install properly to that disk.

dave.parker
2018-04-04 00:30
Turns out the real first disk in this system is /dev/sdb

dave.parker
2018-04-04 00:30
Now it works.

greg
2018-04-04 00:30
cool

dave.parker
2018-04-04 00:30
So frustrating.

shane
2018-04-04 00:31
Hopefully you didn't nuke a useful USB ... :slightly_smiling_face:

dave.parker
2018-04-04 00:31
But turns out, it was hardware, not anything with DRP, I don't think. Unless you guys mount something as USB during the install.

dave.parker
2018-04-04 00:31
If I did, that's too bad. If one of the DC guys left a thumb drive plugged in I'm gonna give him hell tomorrow. :smile:

shane
2018-04-04 00:31
I worked at a company building Hardware appliances once - and our first installer installed the image direct to the /dev/sda device .... which was our source USB thumb drive on the machine ... doh !! we had to bake in guardrails against that ...

dave.parker
2018-04-04 00:32
Ugh fun

dave.parker
2018-04-04 00:32
This would all be moot if Linux had predictable and stable device names.

shane
2018-04-04 00:33
yeah - one of the few really annoying things about Linux - happy the problem is resolved though !! ....and not broken software :face_with_cowboy_hat:

dave.parker
2018-04-04 00:33
Yeah, me too. Thanks for all your help. I can move forward now with the rollout of DRP. :slightly_smiling_face:

shane
2018-04-04 00:34
en masse ... 10s of Thousands !! Forward, March !

cameron.esdaile
2018-04-04 05:34
has joined #community201804

shane
2018-04-04 13:36
@cameron.esdaile $welcome

2018-04-04 13:36
Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

davina
2018-04-04 18:58
has joined #community201804

spector
2018-04-04 22:22
@davina $welcome

2018-04-04 22:22
Digital Rebar welcome information is here > http://rebar.digital/community/welcome.html

amit.handa
2018-04-05 15:39
hi, sledgehammer is not rebooting to install centos-17.04 for one particular VM instance

amit.handa
2018-04-05 15:39
same is working for another VM I just created (virtualbox)

amit.handa
2018-04-05 15:40
on drp, all I see is that sledgehammer is probably doing 'POST /api/v3/machines'

amit.handa
2018-04-05 15:40
repeatedly

amit.handa
2018-04-05 15:40
DRP version : 3.7.3

amit.handa
2018-04-05 15:40
content pack : 1.0.2

amit.handa
2018-04-05 15:40
please help

spector
2018-04-05 15:49
@amit.handa I have reached out to the engineers to look at this

amit.handa
2018-04-05 15:49
Thanks !

shane
2018-04-05 16:01
@amit.handa - content 1.0.2 is _very_ old - could you please upgrade your drp-community-content to the current stable (v1.6.2) - you will need to also load a new Sledgehammer image (`drpcli bootenvs uploadiso sledgehammer`) after updating content ... retry after that - I don't necessarily think your issue will be fixed - but there are a LOT of fixes in content and sledgehammer since you're version

shane
2018-04-05 16:02
you can upgrade your Content from the Contents menu item in the navigation panel

amit.handa
2018-04-05 16:18
thanks shane, let me do it

amit.handa
2018-04-05 16:19
sorry about my wrong info

amit.handa
2018-04-05 16:19
I see that content pack is at v 1.6.2

amit.handa
2018-04-05 16:20
m loading sledgehammer

amit.handa
2018-04-05 16:20
and retrying

amit.handa
2018-04-05 16:31
I see that sledgehammer version is the same as the one I am uploading ... ``` .sledgehammer-9743e672ff33179cd5218d8fe506c03cf2a31d18.tar.part CentOS-7-x86_64-Minimal-1708.iso sledgehammer-9743e672ff33179cd5218d8fe506c03cf2a31d18.tar ```

amit.handa
2018-04-05 16:38
sledgehammer bootenv is showing login prompt. can I login into it and see few logs ? whats the def user/password ?

shane
2018-04-05 16:38
root/rebar1

amit.handa
2018-04-05 16:38
got it

amit.handa
2018-04-05 16:39
it shows validation error: violates unique index Name

amit.handa
2018-04-05 16:40
got

amit.handa
2018-04-05 16:40
resolved

amit.handa
2018-04-05 16:41
sledgehammer was creating machine with the same name as earlier VM (my laptop name)

amit.handa
2018-04-05 16:41
I updated the prev created machine name to a different one

amit.handa
2018-04-05 16:41
and off it went

amit.handa
2018-04-05 16:41
thanks guys

shane
2018-04-05 16:41
excellent

amit.handa
2018-04-05 16:41
but why would it create a m/c with same name ?

amit.handa
2018-04-05 16:42
also, can the same error be shown on the server output

amit.handa
2018-04-05 16:42
I have enabled trace logging

shane
2018-04-05 16:43
default machine name will be a `d`, followed by the MAC address of the machine, with dashes in place of colons, dots, or other chars. did you rename the machine at some point ?

shane
2018-04-05 16:47
here's an example machine name just created: `d08-00-27-40-0e-e7`

amit.handa
2018-04-05 16:47
I didnt. dont remember. however, I have custom kickstart. have created virtualbox VM with networking card in bridge mode

shane
2018-04-05 16:48
ok - check what you do with hostname in the KS ...

shane
2018-04-05 16:48
aren't virtualbox networks fun to play with .... :face_with_rolling_eyes:

aneto
2018-04-05 19:45
hey im running into an issue with a new install. during discovery stage, sledgehammer startup.sh is failing. in the ui logs i get `Failed auth general - info get` and the sledgehammer logs output `Error: CLIENT_ERROR` `Cannot handle content-type text/plain; charset=utf-8`

aneto
2018-04-05 19:46
it looks like an issue on the provision server side. also to note im using the docker container with the latest tag.

zehicle
2018-04-05 19:59
can you hit the API from the provisioned machine? sometimes docker networking creates confusion about the actual path

aneto
2018-04-05 20:00
it was able to fetch the drpcli

aneto
2018-04-05 20:00
and grab a token

zehicle
2018-04-05 20:00
you should be able to test it from the machine and your desktop using `drpcli info get` when you pass in the right credentials

greg
2018-04-05 20:00
@aneto - what version of DRP are you running?

greg
2018-04-05 20:01
There was an intermediate tip bug that caused this problem.

aneto
2018-04-05 20:01
running `drpcli info get` from the pxe booted machine returns `Failed auth general - info get` in the webui logs

greg
2018-04-05 20:01
Updating to latest tip drp and tip content should resolve it.

aneto
2018-04-05 20:01
ahh hmm

aneto
2018-04-05 20:02
i am running latest (tip) from docker hub

aneto
2018-04-05 20:02
so let me blow away my data and start over

aneto
2018-04-05 20:02
maybe there is something in there lingering as i have another test box that doesnt exhibit this issue and is also running latest

greg
2018-04-05 20:03
Docker hub maybe behind.

greg
2018-04-05 20:03
I?m not sure.

greg
2018-04-05 20:05
tip is around 148

aneto
2018-04-05 20:09
currently running tip 148

aneto
2018-04-05 20:10
just blew away all the data and starting fresh to see if it fixes it

aneto
2018-04-05 20:29
had the same issues with tip 148. testing with stable tag now

aneto
2018-04-05 22:34
can i get any direction on what causes `Failed auth general - info get - 192.168.5.252`?

aneto
2018-04-05 23:03
figured it out. the docker container does not include the default Community Content pack. not sure if we were doing this by hand previously without realizing it.

aneto
2018-04-05 23:03
was a very painful issue lol

shane
2018-04-05 23:04
Yes - the assumption is you want a stock just DRP setup - and you may layer in your own Content, or the Community Content, or the RackN licensed content ...

zehicle
2018-04-05 23:05
if you used the quickstart, it would install community content

aneto
2018-04-05 23:05
used the docker container

shane
2018-04-05 23:05
some use cases might even have no content - eg DHCP service only - so you get a really nice clean API driven DHCP server

aneto
2018-04-05 23:05
following quickstart + docker container

aneto
2018-04-05 23:05
bit me in the butt lol

zehicle
2018-04-05 23:05
FWIW, I think the wizard in the UX would show a red X if there was no content.

shane
2018-04-05 23:05
I'll update the docs to mention that if using the Container - to do that

aneto
2018-04-05 23:06
nope. i went to content and it showed i could "upgrade" the community content

aneto
2018-04-05 23:06
because i didnt have it installed :joy:

aneto
2018-04-05 23:06
thanks all!

aneto
2018-04-05 23:06
fantastic app, really digging it over puppet's razor

shane
2018-04-05 23:07
wait until you get in to the cool content / workflow stuff that the newer version can give you .... :slightly_smiling_face:

zehicle
2018-04-05 23:11
working on a videos for showing that and an update on Terraform w/ Workflow!

aneto
2018-04-05 23:22
awesome! when do the videos come out?

spector
2018-04-05 23:27
Rob and team are always creating new videos - you can find the latest at https://www.youtube.com/watch?v=0gbZm0Ltu2c&list=PLXPBeIrpXjfilUi7Qj1Sl0UhjxNRSC7nx

zehicle
2018-04-06 11:57
And here's one with Workflows + Terraform! https://youtu.be/RtuZQHKmd9U

zehicle
2018-04-06 14:39
COMMUNITY REQUEST > if you are enjoying using Digital Rebar, please consider Tweeting at @digitalrebar and giving the project a github star https://github.com/digitalrebar/provision. Spreading the word about the project helps us continue to expand it. Thank you!