Now the last couple of months I’ve again been involved with a Netscaler book project with Packt. This is a more advanced book then my previous book with was a more introduction to Citrix Netscaler.

This new book is called Mastering Netscaler which has more in-depth information regarding load balancing, appfirewall and such.

But… I kinda feel that this book just covers a fragement on what users want to read about when they buy a book about Netscaler.

Therefore in order to get things right, I was thinking about creating a third book about Netscaler which covers all the subjects, stuff you want to read about. Therefore this post is merely for you to give feedback to me

If you could please give me a few senteces about what YOU would want to include in a Netscaler book ? Please drop a comment below this post.

and if you are willing to help me form and maybe contribute to the outline and possibly help me write the book as well that would be great, just send me email to msandbu@gmail.com

Ever wanted a simple way to block pesky IP-adresses which are giving you much unwated traffic on your webservers? Of course there is the possibility to use ACLs but the become cumbersone if we need to add every IP adress to an ACL (They also get unmanageable)

Another option we have is to use pattern sets. Pattern sets are basically an index with different strings which we can then use against an expression to evaluate if they fall within the category or not.

First we need to create the pattern set, under AppExpert –> Pattern Sets (Which is set to include all of those IP-adresses that we don’t want to access our websites.

Next we need an expression which has the ability to extract out the strings and evaluate them against a rule. Go into AppExpert –> Expressions –> Advanced Expressions –>

Create a new expression called CIP, where the expression looks like this

This will allow us when creating a responder policy to add a string in the expression. Next go into URL responder and create a new policy

Now the magic lies within the expression, since we created a custom saved expression we can use that, which basically just says CLIENT_IP_SRC_EQUALS_ANY”(STRING IN THE PATTERN SET nonoIPS) then RESET Connection.

Then we have to bind the policy to either a vServer or globally, and voila. Now we just have to update the pattern set next time we want to block an IP-address. But do not mistake this for an ACL it only block HTTP access.

So the last couple of days I’ve been doing a bit of research on Netscaler and prioritizing traffic based upon where the endpoint is coming from. This is where AppQoE comes in. AppQoE is just a combination of different roles into one role, HTTPDoS, Priority Queuing, and SureConnect.

So what if we have a vServer which is getting pounded by traffic, so how do we prioritize the traffic ? So in AppQoE we have two things. Policies and Actions.

Let’s say that we want to divide traffic into two priority groups. One which are android based devices and another which are windows phone devices. Android based devices are given high priority and Windows phones are given lower priority. There are four priorities we can define in AppQoE. HIGH, NORMAL, LOW and LOWEST. And the Netscaler will process traffic from top to bottom. Meaning that Android traffic is priorited over Windows Phone based traffic.

So I have an example expression here for android devices.

My action looks like this

What is does is basically bind HIGH priority traffic sign to my AppQoE policy, so not much work I have to do here. but next I have to create an AppQoE policy to my Windows Phone users.

My AppQoE action looks like this. Now important to see that the policy queue depth defines how many connenctions needs to be active before it is moved to LOWEST priority. I also have to define max connections, if there are requests over the maximum amount of connenctions I have the Netscaler display a custom wait page ( I choose NS, because then I can use a custom HTML code on the Netscaler, if I choose ACS I can choose another web server for instance.

Now i can attach this policy to a vServer. (NOTE that SureConnect cannot be enabled for a vServer if used with AppQoE)

Now stay tuned for how to setup this with HTTP DOS inorder to protect from HTTP attacks as well with AppQoe.

So this is something I’ve struggeled a bit with in the past, also see it on a couple of forums post on Citrix, and there are as always not so detailed info on how to verify on “WHAT THE HELL IS WRONG WITH THE D*** CONNECTION TO DNS AND LDAP!!!”

So therefore I decided to write this post, since both DNS and LDAP are crucial in adding to the Netscaler.

So lets start with DNS. There are a couple of ways to add DNS on the Netscaler. Either its UDP, TCP or TCP & UDP. Now UDP is the one that is typical used since a default DNS uses UDP, TCP is more for Zone transfers and so on.

So what happens if we add a DNS server using UDP, Well the Netscaler is going to do a ping against the DNS server to see if it is alive (So if ICMP is blocked it will show as DOWN) It will check every 20 seconds to see if it respons on UDP/53. Also imporatant to note that it does use the SNIP address to communicate with the DNS server.

How can we verify that it can do name lookup ? (By default most of the built-in cmdlets like nslookup, dig and so on do not work with Netscaler since it has its own DNS feature built-in, and those cmdlets will only query the local DNS not the external one.

So to test DNS use the command

show dns addRec hostanem

So if we switch from UDP to TCP it will try to use TCP Handshake to verify if it is available, but not going to give use the regular DNS query. So what if we cannot reach the DNS server? Using ping from the cmdlet uses NSIP by default

but with ping in Netscaler we can define a source address (Which we can set to be one of the SNIP addresses.)

ping ip-address –S source-address

If you make a trace file you can also see that it works as it should.

If your SNIP does not have access to the DNS server you need to either define ACLs which allow it to communicate with the DNS server, create a new SNIP which has local access to the DNS server or define a policy based routing which define where the SNIP needs to go to inorder to access the DNS servers.

For instance if I want to setup a specific route for my DNS traffic from my SNIP ( I can setup a PBR) which looks like this (This is a policy route only for ICMP)

After I create the PBR I have to run the command apply pbrs

So that took take of DNS, what about LDAP ? When we setup LDAP servers in Netscaler we have the ability to do retrieve attributes button, great! well almost… it uses the endpoint client IP to retrieve attributes (not the NSIP itself) so it by default uses NSIP. So we can use PING to verify network connectivity. We can also use telnet to verify connectivity since telnet originates from the NSIP.

Shell –-> Telnet

open 192.168.60.1 389 (This can try to connect to the LDAP port 389)

How can you verify it works ? It says connected, if it stands on Trying…. the port is not available. If you want to can change that the Netscaler uses SNIP instead of NSIP, this can be done by setting up a load balanced AD server role, then point the LDAP authentication policy to that vServer.

So this is something I’ve struggeled a bit with in the past, also see it on a couple of forums post on Citrix, and there are as always not so detailed info on how to verify on “WHAT THE HELL IS WRONG WITH THE D*** CONNECTION TO DNS AND LDAP!!!”

So therefore I decided to write this post, since both DNS and LDAP are crucial in adding to the Netscaler.

So lets start with DNS. There are a couple of ways to add DNS on the Netscaler. Either its UDP, TCP or TCP & UDP. Now UDP is the one that is typical used since a default DNS uses UDP, TCP is more for Zone transfers and so on.

So what happens if we add a DNS server using UDP, Well the Netscaler is going to do a ping against the DNS server to see if it is alive (So if ICMP is blocked it will show as DOWN) It will check every 20 seconds to see if it respons on UDP/53. Also imporatant to note that it does use the SNIP address to communicate with the DNS server.

How can we verify that it can do name lookup ? (By default most of the built-in cmdlets like nslookup, dig and so on do not work with Netscaler since it has its own DNS feature built-in, and those cmdlets will only query the local DNS not the external one.

So to test DNS use the command

show dns addRec hostanem

So if we switch from UDP to TCP it will try to use TCP Handshake to verify if it is available, but not going to give use the regular DNS query. So what if we cannot reach the DNS server? Using ping from the cmdlet uses NSIP by default

but with ping in Netscaler we can define a source address (Which we can set to be one of the SNIP addresses.)

ping ip-address –S source-address

If you make a trace file you can also see that it works as it should.

If your SNIP does not have access to the DNS server you need to either define ACLs which allow it to communicate with the DNS server, create a new SNIP which has local access to the DNS server or define a policy based routing which define where the SNIP needs to go to inorder to access the DNS servers.

For instance if I want to setup a specific route for my DNS traffic from my SNIP ( I can setup a PBR) which looks like this (This is a policy route only for ICMP)

After I create the PBR I have to run the command apply pbrs

So that took take of DNS, what about LDAP ? When we setup LDAP servers in Netscaler we have the ability to do retrieve attributes button, great! well almost… it uses the endpoint client IP to retrieve attributes (not the NSIP itself) so it by default uses NSIP. So we can use PING to verify network connectivity. We can also use telnet to verify connectivity since telnet originates from the NSIP.

Shell –-> Telnet

open 192.168.60.1 389 (This can try to connect to the LDAP port 389)

How can you verify it works ? It says connected, if it stands on Trying…. the port is not available. If you want to can change that the Netscaler uses SNIP instead of NSIP, this can be done by setting up a load balanced AD server role, then point the LDAP authentication policy to that vServer.

Now I had the pleasure of talking PCI-DSS compliant XenApp enviroment for a customer. Now after working with it for the last couple of days there are lot of usefull information that I thought I would share.

Now PCI-DSS compliance is needed for any merchant who accepts credit cards for instance an e-commerce size. Or using some sort of application. So this includes all sorts of

* Different procedures for data shredding and logging

* Access control

* Logging and authorization

Now the current PCI-DSS standard is in version 3 –> https://www.pcisecuritystandards.org/documents/PCI_DSS_v3.pdf

The different requirements and assesment procedures can be found in this document. Now Citrix has also created a document for how to setup a compliant XenApp enviroment https://www.citrix.com/content/dam/citrix/en_us/documents/products-solutions/pci-dss-success-achieving-compliance-and-increasing-web-application-availability.pdf you can also find some more information here –> http://www.citrix.com/about/legal/security-compliance/security-standards.html

Now instead of making this post a pure PCI-DSS post I decided to do a more “howto secure yout XenApp enviroment” and what kind of options we have and where a weakness might be.

Now a typical enviroment might looks like this.

So let’s start by exploring the first part of the Citrix infrastructure which is the Netscaler, in a typical enviroment it might be located in the DMZ. Where the front-end firewall has statefull packet inspection to see what traffic goes back and forth. The best way to do a secure setup of Netscaler is one-armed mode and use routing to backend resources and then have another Firewall in between to do deep packet inspection.

First thing we need to do with Netscaler when setting up Netscaler Gateway for instance is to disable SSL 3.0 and default (We should have MPX do to TLS 1.1 and TLS 1.2 but with VPX we are limited to TLS 1.0

Also important to remember th use TRUSTED third party certificates from known vendors, without any known history. Try to avoid SHA-1 based certificates, Citrix now supports SHA256.

Important to setup secure access to management only (since it by default uses http)

This can be done by using SSL profiles which can be attached to the Netscaler Gateway

Also define NONSECURE SSL renegotiation. Also we need to define some TCP parameters. Firstly make sure that TCP SYN Cookie is enabled, this allows for protection against SYN flood attacks and that SYN Spoof Protection is enabled to protect against spoofed SYN packets.

Under HTTP profiles make sure that the Netscaler drops invalid HTTP requests

Make sure that ICA proxy migration is enabled, this makes sure that there is only 1 session at a time established for a user via the Netscaler

Double hop can also be an option if we have multiple DMZ sones or a private and internal zone.

Specify a max login attempts and a timeout value, to make sure that your services aren’t being hammered by a dictonary attack

Change the password for the nsuser!!!

Use an encrypted NTP source which allows for timestamping when logging. (Running at version 4 and above) and also verify that the timezones are running correctly.

Sett up a SNMP monitoring based solution or Command Center to get monitoring information from Netscaler, or use a Syslog as well to get more detailed information. Note that you should use SNMP v3 which gives both Authentication and encryption.

Use LDAPS based authetication against the local active directory server, since LDAP is pure-text based, and use TLS not SSL, and make sure that the Netscaler verifies the server certificate on the LDAP server

It also helps to setup two-factor authentication to provide better protection against user thefts. Make sure that if you are using a two factor authentication vendor that it uses CHAP authentication protocol instead of PAP. Since CHAP is much more secure authentication protocol then PAP

Use NetProfiles to control traffic flow from a particular SNIP to backend resources (This allows for easier management when setting up firewall rules for Access.

Enable ARP spoof validation, so we don’t have any forging ARP requests where the Netscaler is placed (DMZ Zone)

Use a DNSSEC based DNS server, this allows for signed and validated responses. This way you cannot its difficult to hijack a DNS or do MITM on DNS queries.  Note that this requires that you add a nameserver with both TCP and UDP enabled. (Netscaler can function as both a DNSSEC enabled authoritative DNS server and proxy mode for DNSSEC)

If you wish to use Netscaler as an VPN access towards the first DMZ zone, first things you need to do is

1: Update the SWOT library

Create a preauthetnication policy to check for updated antivirus software

Same goes for Patch updates

In most cases try to use the latest firmware, Citrix does release a new Netscaler firmware atleast one every three months which contains bug fixes and security patches as well.

Do not activate enhanced authentication feedback, this enabled hackers to learn more about lockout policies and or if the user is non existant or locked out, disabled and so on.

Set up STA communication using HTTPS (Which requires a valid certificate and that Netscaler trusts the root CA) You also need to setup Storefront using a valid certificate from a trusted Root CA. This should not be a internal PKI root CA since third party vendors have a much higher form a physical security.

If you for some reason cannot use SSL/TLS based communication with backend resources you can use MACSec which is a layer 2 feature which allows for encrypted traffic between nodes on ethernet.

With Netscaler 10.5, Citrix announced the support for SAML Identity Provider on the Netscaler feature. That basically meant that we could in theory use the Netscaler as an identity provider for Office365 / Azure AD. Now I have been trying to reverse engineering the setup since Citrix hasen’t created any documentation regarding the setup.

But now! Citrix recently announced the setup of Netscaler iDP setup for Office365 setup http://support.citrix.com/article/CTX200818

Yay!

on another part Citrix also released a new build of Netscaler VPX (build 56.12) which fixes the CPU utilization bug on Vmware you can see more about the release note here –> http://support.citrix.com/article/CTX200818

And there is also a new PCI DSS report which shows compliance for version 3.

This post, is based upon a session I had for a partner in Norway. How can we use Netscaler to optimize web content?

Let’s face it, the trends are chaging

* Users are becoming less patient (meaging that they demand that applications/services respond quicker. (more then 40% of users drop out if the website takes mroe then 5 – 10 seconds to load) think about how that can affect a WebShop or eCommerce site ?

* More and more mobile traffic (Mobile phones, ipads, laptops. Communicating using 3G/4G or WLAN for that matter) and to that we can add that there is more data being sent across the network as well. Site web applications become more and more complex, with more code and more components as well.

* More demands to availability (Users are demaing that services are available at almost every hour. If we think about it about 5 – 10 years ago, if something was down for about 10 min we didn’t think that much about it, but now ?

* More demands to have secure communication. It wasn’t that long ago that Facebook and Google switched to SSL as default when using their services. With more and more hacking attempts happening online it requires a certain amount of security.

So what can Netscaler do in this equation ?

* Optimizing content with Front-end optimization, Caching and Compression

With the latest 10.5 release, Citrix has made a good jump into web content optimization. With features like lazy loading of images, HTML comment removal, minify JS and inline CSS.  And adding it that after content is being optimized the content can be compressed using gZIP or DEFLATE and sent across the wire (NOTE: that most web servers like Apache and IIS support GZIP and Deflate but it is much more efficient to do this on a dedicated ADC)

And with using Caching to store often accessed data it makes the Netscaler a good web optimization platform.

* Optimizing based upon endpoints.

With the current trend and more users connecting using mobile devices which are using the internett with a wireless conenction. If needs a better way to communicate as well. A god example here is TCP congeston. On a wireless you have a higher amount of packet loss and this requires using for instance TCP Congestion Westwood which is much better suites on wireless connections. Also using features like MTCP (on supported devices) allows for higher troughput as well. And that we can place different TCP settings on different services makes it much more agile.

* High availability

Using features like load balancing and GSLB allows us to deliver a high availabilty and scale solution. And using features like AppQOE to allows us to prioritize traffic in a eCommerce setting might be a valuable asset. Think the scenario if we have a web shop, where most of our buying customers come from a regular PC while most mobile users that are connecting are mostly checking the latest offers. If we ever where to reach our peak in traffic it is useful to prioritize traffic based upon endpoint connecting.

* Secure content

With Netscaler it allows us to create specific SSL profile which we can attack to different services. For instance older applications which are used by everyone might not have the high requirement regarding security, but on the other hand PCI-DSS requires a high level of security. Add to the mix that we can handle many common DDoS attacks on TCP level and on HTTP. We can also use Application firewall which handles many application based attacks, when an own learning feature it can block users which are not following the common user pattern on a website. And that we can specify common URLs which users are not allowed to access.

So to summerize, the Netscaler can be a good component to optimizing and securing traffic, with a lot of exiting stuff happening in the next year! stay tuned.

Now that Citrix released their Netscaler appliance on Azure we have a huge option to do load balancing within the Azure platform. It is also important to think about the other options we have in Azure to do load balancing outside of Netscaler.

Traffic Manager is one of the first options which acts kinda like GSLB which is a DNS based load balancing feature. Which allow us to load balance between endpoints on a cloud service

These services can be spread across different regions. This can either be load balanced based upon performance, round robin or failover.

Problem with DNS based load balancing is that is never gets a full overview of how the traffic is balanced since it basically just spreads the DNS responses. + at Traffic Manager has limited monitoring capabilities since it can only see on HTTP or HTTPS protocol.

We also now have support for nested profiles within Traffic Manager — http://azure.microsoft.com/blog/2014/10/29/new-azure-traffic-manager-nested-profiles/

Now on the other side we have load balancing endpoints on Cloud Services.

When setting up Cloud Services Load balancing we have more option depending on load balancing distribution, ref https://msdn.microsoft.com/library/azure/dn495300

We can have persistency based upon sourceIP or destionationIP for instance, and that we have more monitoring endbpoint based options. This is a more L4 based load balancing approach, which is also a free option to in Azure.

While Netscaler is a complete L4 – L7 load balancing platform which can be used to load balanced based upon many different parameters. Now you can also combine the Netscaler appliance with a HA setup to get the best from both worlds. With also giving you a active/active Netscaler setup within a cloud service http://support.citrix.com/proddocs/topic/netscaler-vpx-10-5/vpx-azure-ha-config-con.html

So when do use the different services ?

If you have a simple web-service which does not require a advanced monitor capabilities and are setup on many different cloud services, use Traffic Manager

If you have a service which are setup within a cloud service which you need to setup a simple load balancing capability on while having low cost, use Load Balanced Endpoints

If you have a service which requires a more advanced service monitoring capabilities and special demand to distribute traffic use Netscaler within a Cloud Serivce.

Now with the ever growing pace of internet traffic, we are being faced with one challenge, an old protocol which is over 15 years old now and is now way any shape to continue in this race, and yes the one I am talking about is the HTTP protocol.

Now over the years, Google has done a great job trying to improve this way of communication with its own protocol called SPDY which uses prioritizing and multiplexing and with transmission headers are sent using GZIP or Deflate. You can read more about SPDY here –> https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3 

Now on the other hand you have the HTTP/2 protocol which the IEFT has worked one, which Google said will replace their own SPDY protocol (http://blog.chromium.org/2015/02/hello-http2-goodbye-spdy-http-is_9.html

You can read more about the RFC on HTTP/2 protocol here –> https://tools.ietf.org/html/rfc7540 but in essence its the same thing as SPDY, since the initial draft of HTTP/2 was based upon SPDY. Another thing that is important to note that communucation with HTTP2 is based upon a binary format since this is much easier to compress, while tradisional HTTP1.1 is upon human readable text. The people over at HTTP Watch did a comparision between tradisional HTTP, HTTP2 and SPDY https://blog.httpwatch.com/2015/01/16/a-simple-performance-comparison-of-https-spdy-and-http2/ and we can see that these new protocols works alot more efficient.

So what else is needed ? We need a web server that supports HTTP/2 or SPDY and we need web clients that support these protocols.

As we can see most web servers are already supported HTTP/2 https://en.wikipedia.org/wiki/HTTP/2#HTTP.2FHTTPS_servers Windows coming with in in Windows Server 2016 and the new version of IIS, and most web browsers support HTTP/2 as well, such as Chrome, Opera, Firefox, Internet Explorer and lastly Microsoft Edge

But for instance Firefox only supports HTTP/2 using TLS 1.2 https://wiki.mozilla.org/Networking/http2 meaning that even if the Netscaler can use HTTP2 over HTTP it will not work with most of the web browsers.

So how do I test that this stuff works ? the simplest thing is to download an addon to Chrome which is called HTTP/2 and SPDY indicator, which basically shows which sites are enabled for HTTP/2 and SPDY and so on.(This extension is available for FireFox as well)

So whenever we are on a site which has HTTP/2 enabled the icon will appear as such

We can also look at the interal table within Chrome by typing chrome://net-internals/#spdy in the address bar.

If this does not work on your chrome version you need to enable SPDY4/HTTP2 within Chrome which can be done using the chrome://flags/#enable-spdy4 flag.

In regards to setting this up on the Netscaler we have to create or alter a HTTP profile, and note this is only available from version 11 and upwards.

And choose enable under the checkbox for HTTP/2, if SPDY is also enabled the following preference is done when communicating with a vServer that has the HTTP profile bound

• HTTP/2 (if enabled in the HTTP profile)
• SPDY (if enabled in the HTTP profile)
• HTTP/1.1

Now in most cases the backend servers are still using HTTP/1.1 In that case the Netscaler works as a proxy and decodes the traffic from the clients to HTTP 1.1 data and restrasmits the data to the backend servers.

It is however important to note that running HTTP/2 on VPX is not supported and hence the clients will fall back to SPDY which is supported on a VPX.

However there are some requirements that are worth noticing on VPX for SPDY as well:

Troubleshooting for SPDY

If SPDY sessions are not enabled even after performing the required steps, check the following conditions.

• If the client is using a Chrome browser, SPDY might not work in some scenarios because Chrome sometimes does not initiate TLS handshake.
• If there is a forward-proxy between the client and the NetScaler appliance, and the forward-proxy doesn’t support SPDY, SPDY sessions might not be enabled.
• NetScaler does not support NPN over TLS 1.1/1.2. To use SPDY, the client should disable TLS1.1/1.2 in the browser.
• Similarly, if the client wants to use SPDY, SSL2/3 must be disabled on the browser.

In the latest release of Netscaler, which came on friday (download link here — https://t.co/nAf5V6NXB8 ) Citrix now enables for Framehawk usage over Remote Access (AKA ICA Proxy) there are only a few small steps needed in order to deploy Framehawk on Citrix

• Enable DTLS on The vServer Gateway (Recipie for enabling DTLS can be found here — https://msandbu.wordpress.com/2015/03/16/enabling-citrix-receiver-audio-over-netscaler-gateway-with-dtls/)
• Unbind / Rebind SSL certificates on the vServer
• Remove any firewall rules for UDP 443

And of course it still requires that the infrastructure has FP2 installed in the backend. So it is going to be interesting to see how this competes with a proper tuned TCP profile on the Netscaler.

So this is something I’ve struggeled a bit with in the past, also see it on a couple of forums post on Citrix, and there are as always not so detailed info on how to verify on “WHAT THE HELL IS WRONG WITH THE D*** CONNECTION TO DNS AND LDAP!!!”

So therefore I decided to write this post, since both DNS and LDAP are crucial in adding to the Netscaler.

So lets start with DNS. There are a couple of ways to add DNS on the Netscaler. Either its UDP, TCP or TCP & UDP. Now UDP is the one that is typical used since a default DNS uses UDP, TCP is more for Zone transfers and so on.

So what happens if we add a DNS server using UDP, Well the Netscaler is going to do a ping against the DNS server to see if it is alive (So if ICMP is blocked it will show as DOWN) It will check every 20 seconds to see if it respons on UDP/53. Also imporatant to note that it does use the SNIP address to communicate with the DNS server.

How can we verify that it can do name lookup ? (By default most of the built-in cmdlets like nslookup, dig and so on do not work with Netscaler since it has its own DNS feature built-in, and those cmdlets will only query the local DNS not the external one.

So to test DNS use the command

show dns addRec hostanem

So if we switch from UDP to TCP it will try to use TCP Handshake to verify if it is available, but not going to give use the regular DNS query. So what if we cannot reach the DNS server? Using ping from the cmdlet uses NSIP by default

but with ping in Netscaler we can define a source address (Which we can set to be one of the SNIP addresses.)

ping ip-address –S source-address

If you make a trace file you can also see that it works as it should.

If your SNIP does not have access to the DNS server you need to either define ACLs which allow it to communicate with the DNS server, create a new SNIP which has local access to the DNS server or define a policy based routing which define where the SNIP needs to go to inorder to access the DNS servers.

For instance if I want to setup a specific route for my DNS traffic from my SNIP ( I can setup a PBR) which looks like this (This is a policy route only for ICMP)

After I create the PBR I have to run the command apply pbrs

So that took take of DNS, what about LDAP ? When we setup LDAP servers in Netscaler we have the ability to do retrieve attributes button, great! well almost… it uses the endpoint client IP to retrieve attributes (not the NSIP itself) so it by default uses NSIP. So we can use PING to verify network connectivity. We can also use telnet to verify connectivity since telnet originates from the NSIP.

Shell –-> Telnet

open 192.168.60.1 389 (This can try to connect to the LDAP port 389)

How can you verify it works ? It says connected, if it stands on Trying…. the port is not available. If you want to can change that the Netscaler uses SNIP instead of NSIP, this can be done by setting up a load balanced AD server role, then point the LDAP authentication policy to that vServer.

So this is something I’ve struggeled a bit with in the past, also see it on a couple of forums post on Citrix, and there are as always not so detailed info on how to verify on “WHAT THE HELL IS WRONG WITH THE D*** CONNECTION TO DNS AND LDAP!!!”

So therefore I decided to write this post, since both DNS and LDAP are crucial in adding to the Netscaler.

So lets start with DNS. There are a couple of ways to add DNS on the Netscaler. Either its UDP, TCP or TCP & UDP. Now UDP is the one that is typical used since a default DNS uses UDP, TCP is more for Zone transfers and so on.

So what happens if we add a DNS server using UDP, Well the Netscaler is going to do a ping against the DNS server to see if it is alive (So if ICMP is blocked it will show as DOWN) It will check every 20 seconds to see if it respons on UDP/53. Also imporatant to note that it does use the SNIP address to communicate with the DNS server.

How can we verify that it can do name lookup ? (By default most of the built-in cmdlets like nslookup, dig and so on do not work with Netscaler since it has its own DNS feature built-in, and those cmdlets will only query the local DNS not the external one.

So to test DNS use the command

show dns addRec hostanem

So if we switch from UDP to TCP it will try to use TCP Handshake to verify if it is available, but not going to give use the regular DNS query. So what if we cannot reach the DNS server? Using ping from the cmdlet uses NSIP by default

but with ping in Netscaler we can define a source address (Which we can set to be one of the SNIP addresses.)

ping ip-address –S source-address

If you make a trace file you can also see that it works as it should.

If your SNIP does not have access to the DNS server you need to either define ACLs which allow it to communicate with the DNS server, create a new SNIP which has local access to the DNS server or define a policy based routing which define where the SNIP needs to go to inorder to access the DNS servers.

For instance if I want to setup a specific route for my DNS traffic from my SNIP ( I can setup a PBR) which looks like this (This is a policy route only for ICMP)

After I create the PBR I have to run the command apply pbrs

So that took take of DNS, what about LDAP ? When we setup LDAP servers in Netscaler we have the ability to do retrieve attributes button, great! well almost… it uses the endpoint client IP to retrieve attributes (not the NSIP itself) so it by default uses NSIP. So we can use PING to verify network connectivity. We can also use telnet to verify connectivity since telnet originates from the NSIP.

Shell –-> Telnet

open 192.168.60.1 389 (This can try to connect to the LDAP port 389)

How can you verify it works ? It says connected, if it stands on Trying…. the port is not available. If you want to can change that the Netscaler uses SNIP instead of NSIP, this can be done by setting up a load balanced AD server role, then point the LDAP authentication policy to that vServer.

Now I had the pleasure of talking PCI-DSS compliant XenApp enviroment for a customer. Now after working with it for the last couple of days there are lot of usefull information that I thought I would share.

Now PCI-DSS compliance is needed for any merchant who accepts credit cards for instance an e-commerce size. Or using some sort of application. So this includes all sorts of

* Different procedures for data shredding and logging

* Access control

* Logging and authorization

Now the current PCI-DSS standard is in version 3 –> https://www.pcisecuritystandards.org/documents/PCI_DSS_v3.pdf

The different requirements and assesment procedures can be found in this document. Now Citrix has also created a document for how to setup a compliant XenApp enviroment https://www.citrix.com/content/dam/citrix/en_us/documents/products-solutions/pci-dss-success-achieving-compliance-and-increasing-web-application-availability.pdf you can also find some more information here –> http://www.citrix.com/about/legal/security-compliance/security-standards.html

Now instead of making this post a pure PCI-DSS post I decided to do a more “howto secure yout XenApp enviroment” and what kind of options we have and where a weakness might be.

Now a typical enviroment might looks like this.

So let’s start by exploring the first part of the Citrix infrastructure which is the Netscaler, in a typical enviroment it might be located in the DMZ. Where the front-end firewall has statefull packet inspection to see what traffic goes back and forth. The best way to do a secure setup of Netscaler is one-armed mode and use routing to backend resources and then have another Firewall in between to do deep packet inspection.

First thing we need to do with Netscaler when setting up Netscaler Gateway for instance is to disable SSL 3.0 and default (We should have MPX do to TLS 1.1 and TLS 1.2 but with VPX we are limited to TLS 1.0

Also important to remember th use TRUSTED third party certificates from known vendors, without any known history. Try to avoid SHA-1 based certificates, Citrix now supports SHA256.

Important to setup secure access to management only (since it by default uses http)

This can be done by using SSL profiles which can be attached to the Netscaler Gateway

Also define NONSECURE SSL renegotiation. Also we need to define some TCP parameters. Firstly make sure that TCP SYN Cookie is enabled, this allows for protection against SYN flood attacks and that SYN Spoof Protection is enabled to protect against spoofed SYN packets.

Under HTTP profiles make sure that the Netscaler drops invalid HTTP requests

Make sure that ICA proxy migration is enabled, this makes sure that there is only 1 session at a time established for a user via the Netscaler

Double hop can also be an option if we have multiple DMZ sones or a private and internal zone.

Specify a max login attempts and a timeout value, to make sure that your services aren’t being hammered by a dictonary attack

Change the password for the nsuser!!!

Use an encrypted NTP source which allows for timestamping when logging. (Running at version 4 and above) and also verify that the timezones are running correctly.

Sett up a SNMP monitoring based solution or Command Center to get monitoring information from Netscaler, or use a Syslog as well to get more detailed information. Note that you should use SNMP v3 which gives both Authentication and encryption.

Use LDAPS based authetication against the local active directory server, since LDAP is pure-text based, and use TLS not SSL, and make sure that the Netscaler verifies the server certificate on the LDAP server

It also helps to setup two-factor authentication to provide better protection against user thefts. Make sure that if you are using a two factor authentication vendor that it uses CHAP authentication protocol instead of PAP. Since CHAP is much more secure authentication protocol then PAP

Use NetProfiles to control traffic flow from a particular SNIP to backend resources (This allows for easier management when setting up firewall rules for Access.

Enable ARP spoof validation, so we don’t have any forging ARP requests where the Netscaler is placed (DMZ Zone)

Use a DNSSEC based DNS server, this allows for signed and validated responses. This way you cannot its difficult to hijack a DNS or do MITM on DNS queries.  Note that this requires that you add a nameserver with both TCP and UDP enabled. (Netscaler can function as both a DNSSEC enabled authoritative DNS server and proxy mode for DNSSEC)

If you wish to use Netscaler as an VPN access towards the first DMZ zone, first things you need to do is

1: Update the SWOT library

Create a preauthetnication policy to check for updated antivirus software

Same goes for Patch updates

In most cases try to use the latest firmware, Citrix does release a new Netscaler firmware atleast one every three months which contains bug fixes and security patches as well.

Do not activate enhanced authentication feedback, this enabled hackers to learn more about lockout policies and or if the user is non existant or locked out, disabled and so on.

Set up STA communication using HTTPS (Which requires a valid certificate and that Netscaler trusts the root CA) You also need to setup Storefront using a valid certificate from a trusted Root CA. This should not be a internal PKI root CA since third party vendors have a much higher form a physical security.

If you for some reason cannot use SSL/TLS based communication with backend resources you can use MACSec which is a layer 2 feature which allows for encrypted traffic between nodes on ethernet.

With Netscaler 10.5, Citrix announced the support for SAML Identity Provider on the Netscaler feature. That basically meant that we could in theory use the Netscaler as an identity provider for Office365 / Azure AD. Now I have been trying to reverse engineering the setup since Citrix hasen’t created any documentation regarding the setup.

But now! Citrix recently announced the setup of Netscaler iDP setup for Office365 setup http://support.citrix.com/article/CTX200818

Yay!

on another part Citrix also released a new build of Netscaler VPX (build 56.12) which fixes the CPU utilization bug on Vmware you can see more about the release note here –> http://support.citrix.com/article/CTX200818

And there is also a new PCI DSS report which shows compliance for version 3.

This post, is based upon a session I had for a partner in Norway. How can we use Netscaler to optimize web content?

Let’s face it, the trends are chaging

* Users are becoming less patient (meaging that they demand that applications/services respond quicker. (more then 40% of users drop out if the website takes mroe then 5 – 10 seconds to load) think about how that can affect a WebShop or eCommerce site ?

* More and more mobile traffic (Mobile phones, ipads, laptops. Communicating using 3G/4G or WLAN for that matter) and to that we can add that there is more data being sent across the network as well. Site web applications become more and more complex, with more code and more components as well.

* More demands to availability (Users are demaing that services are available at almost every hour. If we think about it about 5 – 10 years ago, if something was down for about 10 min we didn’t think that much about it, but now ?

* More demands to have secure communication. It wasn’t that long ago that Facebook and Google switched to SSL as default when using their services. With more and more hacking attempts happening online it requires a certain amount of security.

So what can Netscaler do in this equation ?

* Optimizing content with Front-end optimization, Caching and Compression

With the latest 10.5 release, Citrix has made a good jump into web content optimization. With features like lazy loading of images, HTML comment removal, minify JS and inline CSS.  And adding it that after content is being optimized the content can be compressed using gZIP or DEFLATE and sent across the wire (NOTE: that most web servers like Apache and IIS support GZIP and Deflate but it is much more efficient to do this on a dedicated ADC)

And with using Caching to store often accessed data it makes the Netscaler a good web optimization platform.

* Optimizing based upon endpoints.

With the current trend and more users connecting using mobile devices which are using the internett with a wireless conenction. If needs a better way to communicate as well. A god example here is TCP congeston. On a wireless you have a higher amount of packet loss and this requires using for instance TCP Congestion Westwood which is much better suites on wireless connections. Also using features like MTCP (on supported devices) allows for higher troughput as well. And that we can place different TCP settings on different services makes it much more agile.

* High availability

Using features like load balancing and GSLB allows us to deliver a high availabilty and scale solution. And using features like AppQOE to allows us to prioritize traffic in a eCommerce setting might be a valuable asset. Think the scenario if we have a web shop, where most of our buying customers come from a regular PC while most mobile users that are connecting are mostly checking the latest offers. If we ever where to reach our peak in traffic it is useful to prioritize traffic based upon endpoint connecting.

* Secure content

With Netscaler it allows us to create specific SSL profile which we can attack to different services. For instance older applications which are used by everyone might not have the high requirement regarding security, but on the other hand PCI-DSS requires a high level of security. Add to the mix that we can handle many common DDoS attacks on TCP level and on HTTP. We can also use Application firewall which handles many application based attacks, when an own learning feature it can block users which are not following the common user pattern on a website. And that we can specify common URLs which users are not allowed to access.

So to summerize, the Netscaler can be a good component to optimizing and securing traffic, with a lot of exiting stuff happening in the next year! stay tuned.

Now that Citrix released their Netscaler appliance on Azure we have a huge option to do load balancing within the Azure platform. It is also important to think about the other options we have in Azure to do load balancing outside of Netscaler.

Traffic Manager is one of the first options which acts kinda like GSLB which is a DNS based load balancing feature. Which allow us to load balance between endpoints on a cloud service

These services can be spread across different regions. This can either be load balanced based upon performance, round robin or failover.

Problem with DNS based load balancing is that is never gets a full overview of how the traffic is balanced since it basically just spreads the DNS responses. + at Traffic Manager has limited monitoring capabilities since it can only see on HTTP or HTTPS protocol.

We also now have support for nested profiles within Traffic Manager — http://azure.microsoft.com/blog/2014/10/29/new-azure-traffic-manager-nested-profiles/

Now on the other side we have load balancing endpoints on Cloud Services.

When setting up Cloud Services Load balancing we have more option depending on load balancing distribution, ref https://msdn.microsoft.com/library/azure/dn495300

We can have persistency based upon sourceIP or destionationIP for instance, and that we have more monitoring endbpoint based options. This is a more L4 based load balancing approach, which is also a free option to in Azure.

While Netscaler is a complete L4 – L7 load balancing platform which can be used to load balanced based upon many different parameters. Now you can also combine the Netscaler appliance with a HA setup to get the best from both worlds. With also giving you a active/active Netscaler setup within a cloud service http://support.citrix.com/proddocs/topic/netscaler-vpx-10-5/vpx-azure-ha-config-con.html

So when do use the different services ?

If you have a simple web-service which does not require a advanced monitor capabilities and are setup on many different cloud services, use Traffic Manager

If you have a service which are setup within a cloud service which you need to setup a simple load balancing capability on while having low cost, use Load Balanced Endpoints

If you have a service which requires a more advanced service monitoring capabilities and special demand to distribute traffic use Netscaler within a Cloud Serivce.

Now with the ever growing pace of internet traffic, we are being faced with one challenge, an old protocol which is over 15 years old now and is now way any shape to continue in this race, and yes the one I am talking about is the HTTP protocol.

Now over the years, Google has done a great job trying to improve this way of communication with its own protocol called SPDY which uses prioritizing and multiplexing and with transmission headers are sent using GZIP or Deflate. You can read more about SPDY here –> https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3 

Now on the other hand you have the HTTP/2 protocol which the IEFT has worked one, which Google said will replace their own SPDY protocol (http://blog.chromium.org/2015/02/hello-http2-goodbye-spdy-http-is_9.html

You can read more about the RFC on HTTP/2 protocol here –> https://tools.ietf.org/html/rfc7540 but in essence its the same thing as SPDY, since the initial draft of HTTP/2 was based upon SPDY. Another thing that is important to note that communucation with HTTP2 is based upon a binary format since this is much easier to compress, while tradisional HTTP1.1 is upon human readable text. The people over at HTTP Watch did a comparision between tradisional HTTP, HTTP2 and SPDY https://blog.httpwatch.com/2015/01/16/a-simple-performance-comparison-of-https-spdy-and-http2/ and we can see that these new protocols works alot more efficient.

So what else is needed ? We need a web server that supports HTTP/2 or SPDY and we need web clients that support these protocols.

As we can see most web servers are already supported HTTP/2 https://en.wikipedia.org/wiki/HTTP/2#HTTP.2FHTTPS_servers Windows coming with in in Windows Server 2016 and the new version of IIS, and most web browsers support HTTP/2 as well, such as Chrome, Opera, Firefox, Internet Explorer and lastly Microsoft Edge

But for instance Firefox only supports HTTP/2 using TLS 1.2 https://wiki.mozilla.org/Networking/http2 meaning that even if the Netscaler can use HTTP2 over HTTP it will not work with most of the web browsers.

So how do I test that this stuff works ? the simplest thing is to download an addon to Chrome which is called HTTP/2 and SPDY indicator, which basically shows which sites are enabled for HTTP/2 and SPDY and so on.(This extension is available for FireFox as well)

So whenever we are on a site which has HTTP/2 enabled the icon will appear as such

We can also look at the interal table within Chrome by typing chrome://net-internals/#spdy in the address bar.

If this does not work on your chrome version you need to enable SPDY4/HTTP2 within Chrome which can be done using the chrome://flags/#enable-spdy4 flag.

In regards to setting this up on the Netscaler we have to create or alter a HTTP profile, and note this is only available from version 11 and upwards.

And choose enable under the checkbox for HTTP/2, if SPDY is also enabled the following preference is done when communicating with a vServer that has the HTTP profile bound

• HTTP/2 (if enabled in the HTTP profile)
• SPDY (if enabled in the HTTP profile)
• HTTP/1.1

Now in most cases the backend servers are still using HTTP/1.1 In that case the Netscaler works as a proxy and decodes the traffic from the clients to HTTP 1.1 data and restrasmits the data to the backend servers.

It is however important to note that running HTTP/2 on VPX is not supported and hence the clients will fall back to SPDY which is supported on a VPX.

However there are some requirements that are worth noticing on VPX for SPDY as well:

Troubleshooting for SPDY

If SPDY sessions are not enabled even after performing the required steps, check the following conditions.

• If the client is using a Chrome browser, SPDY might not work in some scenarios because Chrome sometimes does not initiate TLS handshake.
• If there is a forward-proxy between the client and the NetScaler appliance, and the forward-proxy doesn’t support SPDY, SPDY sessions might not be enabled.
• NetScaler does not support NPN over TLS 1.1/1.2. To use SPDY, the client should disable TLS1.1/1.2 in the browser.
• Similarly, if the client wants to use SPDY, SSL2/3 must be disabled on the browser.

In the latest release of Netscaler, which came on friday (download link here — https://t.co/nAf5V6NXB8 ) Citrix now enables for Framehawk usage over Remote Access (AKA ICA Proxy) there are only a few small steps needed in order to deploy Framehawk on Citrix

• Enable DTLS on The vServer Gateway (Recipie for enabling DTLS can be found here — https://msandbu.wordpress.com/2015/03/16/enabling-citrix-receiver-audio-over-netscaler-gateway-with-dtls/)
• Unbind / Rebind SSL certificates on the vServer
• Remove any firewall rules for UDP 443

And of course it still requires that the infrastructure has FP2 installed in the backend. So it is going to be interesting to see how this competes with a proper tuned TCP profile on the Netscaler.

So it is going to be a busy couple of months ahead.. So this sums up what is happening on my part the next months.

28 – 30 October: At the annual Citrix User Group event in Norway, which is a crazy good conference, I will be speaking about using Office365 with Citrix and different integrations and thinks you need to think about there as well http://cugtech.no/?page_id=1031

October-ish: Something I will working for a while, now after I published my Implementing Netscaler VPX book early last year, I got contacted by my publisher earlier this year who wanted a second edition to add the stuff that people thought was missing plus that I wanted to update the content to V11.

Implementing Netscaler VPX second edition contains

• V11 content
• Implementing on Azure, Amazon
• Front-end optimization
• AAA module
• More stuff on troubleshooting and Insight
• More stuff on TCP optimization, HTTP/2 and SSL

+ Cant remember the rest, anyways the Amazon link is here  http://www.amazon.co.uk/Implementing-NetScaler-VPX-TM-Second/dp/1785288989/ref=sr_1_3?ie=UTF8&qid=1442860517&sr=8-3&keywords=netscaler

November-ish: Suprise! This is also something I have been working on for a while, but I cannot take all of the credit. I cant even take half of the credit since I only did about 40% of the work. Earlier this year I got approached by Packt to create another Netscaler book called Mastering Netscaler which was a new book which was supposed to do more of a deep-dive Netscaler book, after months of back and forth with another co-author the book didnt progress as I wanted to…. Luckily I got in touch with another community member which was interested and away we went, now the Mastering Netscaler book is more of a deep-dive book which will be released either in October/November I have nothing to link to yet, but as soon as it is done I wlll be publishing it here. But as I said I only did about 40% of the writing, most of the credit is due to Rick Roetenberg https://twitter.com/rroetenberg great job!