Please clarify the below mentioned questions, 1. Remote leaf switch uplink to router should be connected only via Eth 1/49-1/52 (40G QSFP)or Shall I use any ports from ETh1/1 to Eth1/48 (SFP-10G ports). 2. Please confirm whether the Remote leaf switch is completely zero touch or Do we need to configure any parameters like VLAN, Sub interface, IP address or OSPF config for Remote leaf - WAN router connectivity before sending the switch remote site. Thanks in advance!!!
The remote leaf needs to use an uplink port (port 49 and up), so you can use the 40G port, or you can use a QSA to convert the 40G QSFP form factor to a 1/10G SFP+ form factor. The remote leaf is zero touch. It won't have any config on it since it will not have joined the ACI Fabric yet. We have hardcoded all leafs and spine uplink ports to send an L2 DHCP request (standard DHCP broadcast, nothing exotic) on VLAN-4. When you configure your upstream router sub-interface on vlan 4, and set the DHCP relay address to point to your APIC's internal TEP IP (example 10.0.0.1), the DHCP request will get to APIC. APIC then offers an IP address based on what you configured in the wizard (in my case 10.50.238.x). Along with the DHCP Offer, it will also send a small XML config file that has all the instructions needed for the remote leaf to automatically configure itself to set up the basics to join the fabric. To be clear the WAN router is not zero touch and you should have the VLAN-4 sub-interface and basic routing in place to get back to your physical ACI fabric in the main DC. APIC will have no control or visibility into the WAN router. It just expects you have that VLAN-4 setup waiting for a connection.
Great video!! It's clear and explanation are perfect. In my case, I'm trying to connect 2 remote leaves but when I connect them, they never get IP and remain in "discovering status". The RLeaf got name and OSPF IP interface , and I can contact APIC from this. OSPF is ok on my IPN... I'm quite confuse and I don't know how to debug the situation. Do you have an idea ?
There are a few ways to look at this. First is, we did a lot of improvements in remote leaf in ACI 4.1.2 and later that remove the need for all this VLAN-5 business. I only bring it up to remove that mess from any consideration. Second, if you are stuck in discovery state, the things to check are: 1. Is the remote leaf showing up in the fabric membership tab waiting to be registered? Usually when they show up it means that DHCP relay is working. If they don't show up, can you check that the WAN router that is fronting the remote leaf has a route to the TEP address of the APIC (either the main TEP or the remote routable TEP we added in 4.1.2. Check that the interface on that router sub-int has the DHCP config to send it to APIC. 2. You can also check the APIC DHCP logs to look at the exchange with the remote leaf. You can see the location of the file below. I grep for the serial number of the remote leaf as my search parameter. apic-ams# pwd /var/log/dme/log apic-ams# egrep ISC dhcpd.bin.log | egrep "FDO222309KT" 3. Make sure you using VLAN-4 on your sub-interface on the WAN router fronting the RL. It *must be vlan 4* I usually find that when you cannot discover it is a routing issue between the RL and the APIC that needs to provide the config file in the DHCP exchange. Hope that helps
Great video mate, although one thing is missing in my opinion. Could you please describe the process how RLeaf joins the fabric? I guess it sends dhcp reuqest for both L3 link IP address (to run OSPF later) and VTEP?
Hi Alexei. The RL joins the fabric in the very same method as Multipod. To summarize, the RL has a hard-coded sub-interface tagged on VLAN-4 (but no IP yet). When it sees link up to the local WAN router it will send a DHCP broadcast asking for its address. The WAN router will have a DHCP relay configured, which will send it to the main ACI fabric where APIC is running a fabric-only DHCP service. APIC will send back an IP address that defines the RL link interface, plus a small XML config file that contains config to bring up OSPF. If all that works, the RL will join the fabric.
This is a great Video. What if the IPN network only allows for 1500mtu ? Will this work ? Secondly would it be okay to deploy remote Leaf with less than the minimum bandwidth requirements ? I am assuming that ACI has no way of telling what the actual bandwidth is and it is only a recommendation to use 100Mbps or more ?
Sorry, missed this one. Well, if you only have MTU 1500 you will have fragmentation and very poor service across the IPN. I strongly advise against fragmenting. In fact, I do not even believe TAC would support anything less than MTU of 1550. VxLAN requires that extra 50 bytes of overhead. I also recall seeing others recommend a min of 1600 bytes just to be on the safe side. Of course 9000 bytes is the ideal, if you can make that. Also note that if MTU is less than 9000 bytes, you will also need to adjust the Control Plane MTU in APIC under "System > System Setttings > Control Plane MTU". It is defaulted to 9000. For your second question, you are correct in that ACI has no idea what the WAN bandwidth is, and assumes you have provided the minimums . Something less than the minimum is going to work technically but you might be inviting issues of congestion and delay if that link is seeing a lot of traffic, so not recommended (and maybe not supported by TAC).
Please clarify the below mentioned questions,
1. Remote leaf switch uplink to router should be connected only via Eth 1/49-1/52 (40G QSFP)or Shall I use any ports from ETh1/1 to Eth1/48 (SFP-10G ports).
2. Please confirm whether the Remote leaf switch is completely zero touch or Do we need to configure any parameters like VLAN, Sub interface, IP address or OSPF config for Remote leaf - WAN router connectivity before sending the switch remote site.
Thanks in advance!!!
The remote leaf needs to use an uplink port (port 49 and up), so you can use the 40G port, or you can use a QSA to convert the 40G QSFP form factor to a 1/10G SFP+ form factor.
The remote leaf is zero touch. It won't have any config on it since it will not have joined the ACI Fabric yet. We have hardcoded all leafs and spine uplink ports to send an L2 DHCP request (standard DHCP broadcast, nothing exotic) on VLAN-4. When you configure your upstream router sub-interface on vlan 4, and set the DHCP relay address to point to your APIC's internal TEP IP (example 10.0.0.1), the DHCP request will get to APIC. APIC then offers an IP address based on what you configured in the wizard (in my case 10.50.238.x). Along with the DHCP Offer, it will also send a small XML config file that has all the instructions needed for the remote leaf to automatically configure itself to set up the basics to join the fabric.
To be clear the WAN router is not zero touch and you should have the VLAN-4 sub-interface and basic routing in place to get back to your physical ACI fabric in the main DC. APIC will have no control or visibility into the WAN router. It just expects you have that VLAN-4 setup waiting for a connection.
@@josephezerski5124 Thanks Joseph, I was able to discover my remote leaf and manage it from APIC. Thanks for your reply and detailed video..
Great video!! It's clear and explanation are perfect.
In my case, I'm trying to connect 2 remote leaves but when I connect them, they never get IP and remain in "discovering status". The RLeaf got name and OSPF IP interface , and I can contact APIC from this.
OSPF is ok on my IPN... I'm quite confuse and I don't know how to debug the situation. Do you have an idea ?
There are a few ways to look at this. First is, we did a lot of improvements in remote leaf in ACI 4.1.2 and later that remove the need for all this VLAN-5 business. I only bring it up to remove that mess from any consideration. Second, if you are stuck in discovery state, the things to check are:
1. Is the remote leaf showing up in the fabric membership tab waiting to be registered? Usually when they show up it means that DHCP relay is working. If they don't show up, can you check that the WAN router that is fronting the remote leaf has a route to the TEP address of the APIC (either the main TEP or the remote routable TEP we added in 4.1.2. Check that the interface on that router sub-int has the DHCP config to send it to APIC.
2. You can also check the APIC DHCP logs to look at the exchange with the remote leaf. You can see the location of the file below. I grep for the serial number of the remote leaf as my search parameter.
apic-ams# pwd
/var/log/dme/log
apic-ams# egrep ISC dhcpd.bin.log | egrep "FDO222309KT"
3. Make sure you using VLAN-4 on your sub-interface on the WAN router fronting the RL. It *must be vlan 4*
I usually find that when you cannot discover it is a routing issue between the RL and the APIC that needs to provide the config file in the DHCP exchange.
Hope that helps
Great video mate, although one thing is missing in my opinion. Could you please describe the process how RLeaf joins the fabric? I guess it sends dhcp reuqest for both L3 link IP address (to run OSPF later) and VTEP?
Hi Alexei. The RL joins the fabric in the very same method as Multipod. To summarize, the RL has a hard-coded sub-interface tagged on VLAN-4 (but no IP yet). When it sees link up to the local WAN router it will send a DHCP broadcast asking for its address. The WAN router will have a DHCP relay configured, which will send it to the main ACI fabric where APIC is running a fabric-only DHCP service. APIC will send back an IP address that defines the RL link interface, plus a small XML config file that contains config to bring up OSPF. If all that works, the RL will join the fabric.
@@josephezerski5124 thank you very much, that was really clear explanation!
This is a great Video. What if the IPN network only allows for 1500mtu ? Will this work ? Secondly would it be okay to deploy remote Leaf with less than the minimum bandwidth requirements ? I am assuming that ACI has no way of telling what the actual bandwidth is and it is only a recommendation to use 100Mbps or more ?
Sorry, missed this one. Well, if you only have MTU 1500 you will have fragmentation and very poor service across the IPN. I strongly advise against fragmenting. In fact, I do not even believe TAC would support anything less than MTU of 1550. VxLAN requires that extra 50 bytes of overhead. I also recall seeing others recommend a min of 1600 bytes just to be on the safe side. Of course 9000 bytes is the ideal, if you can make that. Also note that if MTU is less than 9000 bytes, you will also need to adjust the Control Plane MTU in APIC under "System > System Setttings > Control Plane MTU". It is defaulted to 9000.
For your second question, you are correct in that ACI has no idea what the WAN bandwidth is, and assumes you have provided the minimums . Something less than the minimum is going to work technically but you might be inviting issues of congestion and delay if that link is seeing a lot of traffic, so not recommended (and maybe not supported by TAC).