August 1, 2016 Leave a comment
There are times you really need to take things beyond the “file new” experience and implement a more advanced scenario. And with these opportunities, there are times you realize that what you need likely isn’t a “one off” kind of thing. There are larger implications to what you need that can help solve a myriad of problems. This is the story of one these scenarios.
I was recently working with a partner as they explored Service Fabric . They liked what they saw, but there was a “but” (there almost always is). This partner is in the government space, and one of the requirements they had is that all public facing services are isolated and secured from any “back end” services (in a DMZ). If you’ve been doing IT for any length of time, this shouldn’t come as news. But the question they had for me was how to do this with Service Fabric.
There were a couple ways to address this that immediately came to mind. We could deploy the front end web application as an Azure Web App , hosted in an App Service Environment that was joined to the same VNet as the Service Fabric Cluster. We could also set up two Service Fabric clusters, again joined by a single VNet. The issue with both of these is that the front and back ends of the solution would need to be deployed and managed separately. Not a huge deal admittedly. But this did complicate the provisioning and deployment processes a bit, as well as seemed to run counter to the idea of a Service Fabric “application”, composed of multiple services as a single entity. I was fortunate that I had previously engaged my friend and colleague Kal to bring his considerable Service Fabric experience into play with this partner, and he suggested a third option, one we all found fairly intriguing.
A Service Fabric cluster has Node Types which are directly related to VM Scale Sets . Taking advantage of this, we could place different node types into different subnets and place Network Security Groups (NSGs) on the subnets to provide the level of isolation the partner required. We would then use Placement Constraints to ensure that the services within an application are only hosted in the proper subnet by using constraints specific to the node type, or types, in that subnet.
We ran the idea by Mark Fussell , the lead Project Manager of the Service Fabric team. As we talked, we realized that folks had secured a cluster from all external access, but there didn’t appear to be a public, previously documented version of what we were proposing. Mark was supportive of the idea, and even offered up that in some of the “larger” Service Fabric clusters, the placement constraint approach has been used to ensure that the services that make up the Service Fabric Cluster remain isolated from those that comprise the applications deployed within it.
Our mission clear, I set to work! We were going to create a Azure Resource Manager template to create our “DMZ’d Service Fabric Cluster”.
Network TopologyThe first step was to create the overall network topology.
We have the front end subnet, which has a public load balancer that would handle traffic from the internet via a load balancer. There is a back end subnet with an internal load balancer that does not allow any connections from outside of the virtual network (using a private IP). Finally, we have a management subnet that contains the cluster services, including the web portal (on port 19080) and TCP client API (19000). For good measure, we’re also going to toss an RDP jump box into this subnet so if something goes wrong with any of the nodes in the cluster, we can remote in and troubleshoot (something that I used the heck out of while crafting this template).
With this in place, we then define the VM Scale Sets, and bind their network configurations to the proper subnets as follows:
"networkInterfaceConfigurations": [{
"name": "[variables('nodesMgmnt')['nicName']]",
"properties": {
"ipConfigurations": [
{
"name": "[concat(variables('nodesMgmnt')['nicName'],'-',0)]",
"properties": {
"loadBalancerBackendAddressPools": [
{
"id": "[variables('lbMgmnt')['PoolID']]"
}
],
"subnet": {
"id": "[variables('subnetManagement')['Ref']]"
}
}
}
],
"primary": true
}
}
]
With the VM Scale Sets in place, then we moved on to the Service Fabric Cluster to define each Node Type. Here’s the cluster node type definition for the management subnet node type.
{"name": "[variables('nodesMgmnt')['TypeName']]",
"applicationPorts": {
"endPort": "[variables('svcFabCluster')['applicationEndPort']]",
"startPort": "[variables('svcFabCluster')['applicationStartPort']]"
},
"clientConnectionEndpointPort": "[variables('svcFabCluster')['tcpGatewayPort']]",
"durabilityLevel": "Bronze",
"ephemeralPorts": {
"endPort": "[variables('svcFabCluster')['ephemeralEndPort']]",
"startPort": "[variables('svcFabCluster')['ephemeralStartPort']]"
},
"httpGatewayEndpointPort": "[variables('svcFabCluster')['httpGatewayPort']]",
"isPrimary": true,
"placementProperties": {
"isDMZ": "false"
},
"vmInstanceCount": "[variables('nodesMgmnt')['capacity']]"
}
The “name” of this Node Type, must match the name of a VM Scale Set, that’s how the two get wired together. Since this sample is for our “management” node type, it would also be the only one with the isPrimary property set too true.
At This point, we debugged the template and made sure the cluster to ensure it was valid and the cluster would come up “green”. The next (and harder step) is to start securing the cluster.
Note:If you create a cluster via the Azure portal with multiple node types, each node type will get its own subnet. However, we were after a reusable ARM template so we had to configure things ourselves.
Network SecurityUnfortunately, when we set out to create this, there wasn’t much publicly available on the ports that were needed within a fabric cluster. So we had to do some guesswork, some heavy digging, as well as make a wishes for some good luck. So in this section I’m hoping to lay out some of what we learned to save others the effort.
First off, we started by blocking all inbound connections on the three subnets. I then opened ports 19080 (used by the Service Fabric web portal) and 19000 (used by the Fabric Client and Powershell) for the “management” subnet so I could interact wit