Cisco DCNM – POAP-2-POAPFAILURE: POAP boot file download failed.

Cisco Nexus 9K POAP Error message

2018 Oct  8 18:49:53 switch %$ VDC-1 %$ %POAP-2-POAP_DHCP_DISCOVER_START: [SERIALNUMBER-MACADDRESS] - POAP DHCP Discover phase started
2018 Oct  8 18:50:15 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [SERIALNUMBER-MACADDRESS] - Using DHCP, information received over mgmt0 from 192.168.1.8
2018 Oct  8 18:50:15 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [SERIALNUMBER-MACADDRESS] - Assigned IP address: 192.168.1.254
2018 Oct  8 18:50:15 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [SERIALNUMBER-MACADDRESS] - Netmask: 255.255.254.0
2018 Oct  8 18:50:15 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [SERIALNUMBER-MACADDRESS] - DNS Server: 192.168.1.23
2018 Oct  8 18:50:15 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [SERIALNUMBER-MACADDRESS] - Default Gateway: 192.168.1.1
2018 Oct  8 18:50:15 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [SERIALNUMBER-MACADDRESS] - Script Server: 192.168.1.8
2018 Oct  8 18:50:15 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [SERIALNUMBER-MACADDRESS] - Script Name: poap_dcnm.py
2018 Oct  8 18:50:27 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [SERIALNUMBER-MACADDRESS] - poap_dhcp_intf_ac_action_configuration_success: the script download string is [copy tftp://192.168.1.8/poap_dcnm.py bootflash:scr
2018 Oct  8 18:50:27 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [SERIALNUMBER-MACADDRESS] - The POAP Script download has started
2018 Oct  8 18:50:27 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [SERIALNUMBER-MACADDRESS] - The POAP Script is being downloaded from [copy tftp://192.168.1.8/poap_dcnm.py bootflash:scripts/script.sh vrf management ]
2018 Oct  8 18:50:53 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: [SERIALNUMBER-MACADDRESS] - POAP boot file download failed.
2018 Oct  8 18:50:54 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: [SERIALNUMBER-MACADDRESS] - POAP DHCP discover phase failed
2018 Oct  8 18:51:00 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [SERIALNUMBER-MACADDRESS] - USB Initializing Success
2018 Oct  8 18:51:00 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [SERIALNUMBER-MACADDRESS] - USB disk not detected

Troubleshooting According to Cisco

This is suggested from Ciscos Website:

Troubleshooting PowerOn Auto Provisioning

But it is of no help.

• Enable Guestshell
  • guestshell enable

switch(config)# guestshell enable
switch(config)# 2018 Oct  8 18:37:11 switch %$ VDC-1 %$ %VMAN-2-INSTALL_STATE: Installing virtual service 'guestshell+' 
2018 Oct  8 18:37:38 switch %$ VDC-1 %$ %VMAN-2-INSTALL_STATE: Install success virtual service 'guestshell+'; Activating
2018 Oct  8 18:37:38 switch %$ VDC-1 %$ %VMAN-2-ACTIVATION_STATE: Activating virtual service 'guestshell+'  
2018 Oct  8 18:38:01 switch %$ VDC-1 %$ %VMAN-2-ACTIVATION_STATE: Successfully activated virtual service 'guestshell+'  
2018 Oct  8 18:38:01 switch %$ VDC-1 %$ %VMAN-2-GUESTSHELL_ENABLED: The guest shell has been enabled. The command 'guestshell' may be used to access it, 'guestshell destroy' to remove it.

• Look for log error messages in /bootflash with the date that you are of the file you are looking for it is prepended with the date in the following format:
  • yyyymmdd

[admin@guestshell ~]$ tail /bootflash/20181005_172211_poap_2027_init.log 

• An example Error message:

Fri Oct  5 22:21:37 2018:poap_dhcp_intf_ac_action_config_interface_select: Failed in the interface selection to send DHCPREQUEST for interface 1a005e00

Troubleshooting According to Me

Test to see if the switch can get to the TFTP Server

• Exit POAP by typing pressing enter
• Then type yes

Abort Power On Auto Provisioning [yes - continue with normal setup, skip - bypass password and basic configuration, no - continue with Power On Auto Provisioning] (yes/skip/no)[no]: yes
Disabling POAP.......Disabling POAP

• Once in the Switch, configure the Mgmt0 port with the IP address that was being assigned in your POAP logs.

conf t
int management 0/0
ip address 192.168.1.254/23

• Now attempt the command that failed

switch#copy tftp://192.168.1.8/poap_dcnm.py bootflash:scr
Trying to connect to tftp server......
Connection to Server Established.
TFTP get operation failed:Connection timed out   

• From this example, we see that the TFTP server is not even being allowing connections.
• This topology does not have a firewall in-between the two nodes, but if you do, check the firewall settings and make sure it is not being blocked.

Verify the TFTP Server is Listening on the Correct Interface or IP Address

• SSH as root into the DCNM

ssh -l root dcnm-IPaddress

• Run grep bind /etc/xinetd.d/tftp to see if the IP address it is configured to listen on is correct.

[root@lab-dcnm dcnm]# grep bind /etc/xinetd.d/tftp
    bind = 192.168.0.24

• In our case, it is not listening on the correct IP address or subnet, so lets change it.
• nano /etc/xinetd.d/tftp
• Change the bind = <ip address> to the correct IP address
• Press ctrl + x to exit
• press y to save
• Note its running now

[root@lab-dcnm dcnm]# netstat -a | grep tftp
udp        0      0 lab-dcnm-ooo.net.s:tftp 0.0.0.0:*                          
udp6       0      0 [::]:tftp               [::]:*   

• For some reason I had to enable and start xinetd

systemctl start xinetd
systemctl enable xinetd

• Then had to start xinetd tftp

• systemctl start xinetd tftp
          systemctl status xinetd tftp
          

output

[root@lab-dcnm dcnm]# systemctl start xinetd tftp
[root@lab-dcnm dcnm]# systemctl status xinetd tftp
● xinetd.service - Xinetd A Powerful Replacement For Inetd
   Loaded: loaded (/usr/lib/systemd/system/xinetd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-10-04 08:33:35 PDT; 4 days ago
 Main PID: 15991 (xinetd)
    Tasks: 1
   Memory: 260.0K
   CGroup: /system.slice/xinetd.service
           └─15991 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid

• Stop TFTP systemctl stop xinetd tftp

[root@lab-dcnm dcnm]# systemctl stop xinetd tftp
Warning: Stopping tftp.service, but it can still be activated by:
  tftp.socket
[root@lab-dcnm dcnm]# netstat -a | grep tftp
udp6       0      0 [::]:tftp               [::]:*  

• Start TFTP systemctl start xinetd tftp

[root@lab-dcnm dcnm]# systemctl status xinetd tftp
● xinetd.service - Xinetd A Powerful Replacement For Inetd
   Loaded: loaded (/usr/lib/systemd/system/xinetd.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-10-08 13:07:38 PDT; 7s ago
  Process: 19170 ExecStart=/usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid $EXTRAOPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 19172 (xinetd)
    Tasks: 1
   Memory: 256.0K
   CGroup: /system.slice/xinetd.service
           └─19172 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid

• Now we test the tftp connection again and it works

switch# copy tftp://192.168.1.9/poap_dcnm.py bootflash:scripts/script.sh vrf management
Trying to connect to tftp server......
Connection to Server Established.
TFTP get operation was successful                                                   
Copy complete, now saving to disk (please wait)...
Copy complete.