Fibre Channel Over Ethernet: From Fee To Free 87
alphadogg writes "With demand for Fiber Channel over Ethernet (FCoE) more sluggish than vendors had hoped, 10 Gigabit Ethernet switch and adapter makers are making it available for free. FCoE is a standard driven largely by Cisco to converge customers' data center LAN and storage fabrics with 10G Ethernet. Industry heavyweights Intel and Brocade are among those now giving away FCoE capabilities. There are several factors prompting vendors to slash FCoE prices or stop charging for it altogether, including market indifference; technological immaturity; competing alternatives, such as virtualized Fibre Channel and Ethernet I/O; the recession; and vendors looking to drive switch volumes. 'When FCoE first came out there used to be a fairly large price premium,' says Alan Weckel, director of Dell'Oro Group. 'Cisco had to give it away for free to drive switch volumes. Users were not adopting as rapidly as thought or that Cisco had hoped for.'"
CSCO forward guidance is low (Score:2, Interesting)
Re: (Score:2)
I'd still go for iSCSI, by the way. Even cheaper, and routable if need be. For truly mission-critical stuff: use DAS.
Re: (Score:2)
I think we are starting to see a trend....anything that people do not need to upgrade, they do not, who needs a octo core computer when surfing the web, or windows 7 when xp does the job just fine, or a regular 10/100 router, when the 10gb are available... in the end, it boils down to the computer sector was overpriced for many eyars, and now we are seeing the true value of things.....gone are the days that laptops used to sell for about 2000$ when bestbuy has trouble moving the 250$ netbooks which are twic
Re: (Score:2)
Too late (Score:4, Insightful)
As network fabric bandwidth continued to increase and latency decrease, FCoE appeared to be a last ditch effort to plug the steady trickle of customers from the highly expensive FC over to the much cheaper to deploy iSCSI. I'm sure the thinking was that by making it routable and with the same semantics as existing FC installs, it could accomplish that task. However, I'm also thinking that in most situations, where there's little to distinguish between iSCSI and FCoE other than the now almost commonplace on-NIC hardware iSCSI acceleration, it's a case of too late.
Re:Too late (Score:5, Informative)
Actually, FC isn't routable. FC over *ethernet* has no ip and no provisions to span gateways. The *theory* is that FCoE has fewer layers allowing for higher performance, but it's rare for that difference to be realized in cheap ethernet fabrics (the whole point of FCo*E*) and even rarer to matter relative to storage device performance limitations. iSCSI is much easier to manage with fewer limitations and gets some nice things from being over TCP whether FCoE people will admit it or not.
When FCoE first came to market, vendors had dollarsigns in their eyes with thoughts of extorting customers with FC pricing strategies using 'just' ethernet. You saw people trying to do per-port FC enablement licensing BS and other stuff unheard of in ethernet land.
If FCoE is going to exist long term, it will be as a 'freebie' alternative to iSCSI or as a convenience to build large SANs without a lot of FC switches and HBAs but using existing FC enclosures.
Re: (Score:3)
Re:Too late (Score:4, Interesting)
I think he meant that FC is not routable as the standard IP protocol is unless you buy expensive and proprietary solutions (as you call it, a hack).
iSCSI has the advantage of being able to sustain a packet loss while FCoE can't stand it. FCoE is thus only possible over a small fabric (eg. 1 stack of dedicated switches) while iSCSI can be mixed with other traffic without sustaining any issues. Off course, some people using iSCSI can't sustain any packet losses either (because of latency - eg. streaming video and live video editing) and don't understand that Ethernet is not built for that kind of load - network engineers don't care about packet losses and hope the transport layer will fix them, storage engineers can't have any packet losses and the transport layer relies on it.
That being said, FCoE is similar to ATAoE, never widespread because of it's iSCSI cousin and those that ended up using it, might as well just used a true FC (or Infiniband) fabric.
Re: (Score:3)
network engineers don't care about packet losses and hope the transport layer will fix them
That is a bit of a generalization don't you think? Excessive packet loss is death to any real network, which is why there has been a lot of effort to do various forms of explicit congestion notification instead.
On the Internet, that may be hard, but on a local network it is much easier. Some switches even have ECN marking these days, which is a far superior solution to avoiding loss through buffer bloat.
Re: (Score:2)
This is really the truth.
Most TCP stacks have fairly complex algorithms to avoid, manage, and recover from congestion. Furthermore, with technologies such as sliding windows, TCP allows for rather good scaling once you leave the single switch. This in turn means iSCSI can better scale and intermix with other traffic without catastrophic issue. Even better, loss of frames results in increased latency rather than loss of data.
So while iSCSI is in fact a fatter, heavier stack, that's also why there are dedicat
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
FC over *ethernet* has no ip and no provisions to span gateways
That is why they invented TRILL [wikipedia.org]. Link state routing at layer 2.
Re: (Score:2)
I would have considered that just enhanced switching (it solves a lot of topology problems with large layer 2 ethernet networks, but still would have issues with broadcast in widespread deployment). I know a lot of the terminology says 'routing', but it just seems more logically close to sane enhancements to switching than routing. I recognize at some point the distinction between 'switching' and 'routing' gets blurry.
Re: (Score:1)
FCoE is a L2 protocol, it's not routable. You may be thinking of FCIP. FCoE would've made more sense as an access layer technology early on if some of the supporting standards (TRILL, etc) had been ready at the same time FCoE devices started hitting the market.
Re: (Score:3)
FC is expensive, compared to what?
Unless you're using the (mostly shit) iSCSI software initiators, you'll be using iSCSI hardware initiators on 10gbit Ethernet - which is hardly cheap.
Do the math: to get > 1Gbit/s on your fabric-side links, you need to spend a copious amount if you're going 10gigE. Not only are the switches ungodly expensive compared to 1Gbit, but they're expensive compared to pretty much everything else - Infiniband or Fibrechannel. When it comes down to it, the biggest thing 10gig Ethe
Re: (Score:2)
Do the math: to get > 1Gbit/s on your fabric-side links, you need to spend a copious amount if you're going 10gigE. Not only are the switches ungodly expensive compared to 1Gbit
At work the iSCSI chassis we've been buying have 4x 1Gb ports which bind to make a 4 Gb pipe (the initiator has to be capable). Not 10GigE but much cheaper than a 10 Gb-ready switch.
'course, if you need 10 Gbit, you need 10 Gbit but this is a nice trade-off if super-performance isn't critical.
Re: (Score:2)
Does that actually give you 4gbit throughput to any host across a single data stream, or is like most link aggregation schemes in that it just spreads concurrent sessions across multiple physical links, but each session is limited by the bandwidth of a single physical link?
Re: (Score:2)
(standing on bus, think I got it all...
Re: (Score:2)
Sounds like the Equallogic model. I've always wondered what kind of actual bandwidth you get and how it gets spread out, I can get never get a straight answer from our Equallogic rep and the management software doesn't really give you any idea beyond simple bytecounts per physical interface.
In most of the setups I've been around (recent model PS4000 and PS6000s) using the ESX 4.1 sofiware iSCSI initiator, we see real disk throughput top out around 60 MByte/sec in any given VM with basically no other ongoin
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Plus you can get 48 10Gb ports in 1U. No one else in the industry can touch that density
Not entirely true. Arista has little secret sauce. They're using merchant silicon. Stay tuned for the plethora of switches about to be released based on the Broadcom Trident ASIC [broadcom.com] (64 10 GigE switch on chip, manifesting itself as either 64 10 GigE or 48 10 GigE + 4 40 GigE). Some (like Force10's [force10networks.com]) are already announced. The differentiator in 10 GigE ToR switching is quickly becoming software.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Seriously? We've experimented with iSCSI here and, in our experience, the performance leaves a lot to be desired. I have to assume that's because of the IP overhead, and I'd expect something like HyperSCSI or AoE to perform significantly better. (Why people would rather use iSCSI than something like those is beyond me.)
We're heading in the SAS direction, instea
Re: (Score:2)
What do you mean you're using SAS? Are you using DAS and then something like NFS? That doesn't make a lot of sense, SAS isn't really an alternative to a SAN.
And still, no one buys it. (Score:4, Interesting)
FCoE...
A solution in search of a problem. 10GbE ethernet is really very nice. FC (and FCoE included) have a history of poor vender interop.
So by using FCoE you get the worst of both worlds, 10GbE with vendor lockin at the storage level....
So... NFS anyone (or I guess iScsi)?
Only time i've ever used FCoE was as a WAN tunnel link for asynch rep.... not seeing any other value for this anytime soon.
nobody buys 10GbE either... (Score:3)
10GbE ethernet is really very nice.
Too bad you can't really buy it, and it's insanely expensive, with per-port costs in the hundreds of dollars range. Lots of choices for adapters (which are also insanely expensive)....but I went looking for a 10GbE switch for our small-ish server room for some of our higher bandwidth systems that easily saturate gigabit ethernet...and came up very short in terms of selection. The vast majority of the market consists of switches with 1-2 10GbE uplink ports. That's sli
Re: (Score:2)
Re: (Score:2)
Thats $350 per port, I think he was right about "insanely expensive".
Re: (Score:2)
Re: (Score:2)
When we are talking a t5240 or a M5000, so what? Fraction of the total cost, especial for an Oracle RAC cluster.
Even @$1000 per port, gimme 2, I will use the bandwidth. Hell, gimme 4 on the M5000, eventually they will use it.
Problem is for CISCO, companies that need that bandwidth are too few to drive their revenue model.
Re: (Score:2)
Yeah, but equivalent to 240 ports of GbE. Or equivalent to an $870 24-port 1Gb. That's about the going rate for a decent L3 Gb switch, even HPs. And that's a lot of cables saved, lot of server NICs, etc. 10Gb is really the only way if you have dense stuff like blades or SAN nodes that can push a lot of bits. Obviously. But it's a little more expensive, but only a little. $8k is not really that much when you look at what 50 copies of windows is or *shudder* 50 new iMacs. If you're a medium-sized shop
Re: (Score:2)
That $350 doesn't include transceivers which you need at both ends.
I would think about $4k/port is more realistic for an average install (which won't be using Linksys).
Re: (Score:2)
$367 [provantage.com] for a 10GbE port, $454 [provantage.com] optics/port, $691 Intel 10GbE NIC [cdwg.com] (dual port too)
Total: $1,512/port
So unless you can build out 1Gb for less than $150/port (and have enough space for 10x the ports!) then 10GbE starts looking pretty attractive. But it depends on the size of the isntall, if we start considering a core/distribution/access architecture and including all the upstream ports, etc, it could get incredibly expensive. You could also include cost to install, configure, manage, etc. Bu
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:1)
I really can't understand why anyone would do this FCoE?
Its a crappy protocol to start with, the only useful part of it was the fact that fiber was faster than copper at the time for any reasonable price, now thats no longer true. Fast enough copper is WAY cheaper than fiber and with none of the trouble.
FCoE to me makes about as much sense as PPPoE, when you start talking like that you've clearly very little actual understanding of what you are doing. You can come up with some reasons as to why you might
Re: (Score:2)
It's ISCSI that is a great cheap SAN protocol. FC still has some use to it that ISCSI can't beat, but for most stuff, ISCSI is awesome.
I just don't see a lot of reason to bridge the two, unless you are transitioning in a very very large environment.
Look at Cisco's switches that try to bring FC and 10GbE together -- they suck! No layer 3 support, and the latency is horrible compared with competitors.
People will complain about the overhead that the network stack adds to it, or latency or junk, but really, i
Re: (Score:2)
Re: (Score:1)
Which solution are you referring to when you say FCoE over a WAN?
I'm not aware of any products like that on the market.
You may be referring to the FC over IP which is actually in use quite a lot for years now in situations where native FC either isn't technically possible or would be way too expensive.
FCoE has practically nothing to do with FC over IP.
Planned Obsolescence (Score:3)
Duh (Score:3)
Re:Duh (Score:4, Insightful)
As to your claims that 10Gb FCoE is slower than 8Gb FC for throughput that's rubbish as the framing overhead for FCoE is nowhere near 20% and they are both lossless protocols, for latency it may or may not be true depending on implementation.
Sorry, just not true... (Score:2)
I don't know what the latest iSCSI over 10Gb ethernet scores are but...
8Gb FCP/FC runs at ~8Gbs
10Gb FCoE runs at ~9.7Gbs
Re: (Score:2)
You've gotten 9.8Gbs with FCoE =) hehe, rock on.
Sorry, my bad on the 8G, you are correct. I usually don't get as high as 870MB/s more like 810-820, but my Gbs conversion was WAY off. Thanks for the correction!!!
Re: (Score:1)
Meh, we're already breaking those speeds with local data buses, what's the point of this when we'll be getting it wirelessly soon enough?
Oh, vendor lock-in. Duh.
Ignore me.
Re: (Score:3)
You also had 1/10th the bandwidth and twice as much cabling to each server, higher power draw, more rack space required and more devices to manage.
Re: (Score:2)
Re: (Score:2)
Too late for us IMHO (Score:1)
At work a few years ago we were looking at FCoE but it was huge coin. We opted for iSCSI and haven't looked back. Our gear doesn't have to be super-zippy so we started with 16 drive iSCSI->SATA2 chassis in RAID 6 w/ hotspare. We can bind 4 GigE channels for decent throughput. Not 10 Gb speed but great for our purposes. YMMV.
Forgive my ignorance... (Score:2)
Does the switch have firmware that actually dedicates processor time to blocking FCoE traffic unless you pay the man(and is a license fee for "UDP over ethernet" or "HTTP over ethernet" the next brainwave from Cisco?), or is the "over ethernet" a marketing exaggeration, and there are actually certain non-ethernet features that the switch must support in order to handle FC "over ethernet"?
Re: (Score:3)
One is most 'FCoE' equipment had FC and ethernet ports, so there was a hardware difference.
Another is FCoE generally means the ethernet switch has some FC layer management features (e.g. looking at WWN, zoning, etc). I think this is the *key* priced modification for most FCoE equipment without FC ports. Basically making a way of dealing with the switches exactly the way storage admins are accustomed to dealing with SAN switches.
Finally, there are some layer 2 features considered essentially mandatory for
Re:Forgive my ignorance... (Score:4, Informative)
Re: (Score:2)
Apparently the latter.
Since classical Ethernet has no flow control, unlike Fibre Channel, FCoE requires enhancements to the Ethernet standard to support a flow control mechanism (this prevents frame loss). . . .
Fibre Channel required three primary extensions to deliver the capabilities of Fibre Channel over Ethernet networks:
--Wikipedia
It's all about what the swich is capable of (Score:5, Informative)
Your plain-vanilla 10GbE switch does not have the flow-control bits required to make Ethernet lossless; without essentially lossless traffic, SCSI/FC perf goes in the dumpster. (0.03% packet loss == approx. 50% performance cut.)
In addition, there must be at least one switch in the VLAN that can provide FC services, such as zoning, address assignment, name services, etc.
Re: (Score:2)
(0.03% packet loss == approx. 50% performance cut.)
Do you have a link to back that up? That would be very interesting if true.
Re: (Score:2)
Little early... (Score:2)
It's a little early to call the death of FCoE. We still can't really do a true FCoE environment. The firmware to enable multi-hop FCoE on switches is just now starting to ship. Up to now all we've done is single-hop where the storage is directly connected to the same switch as the end devices...which is not scalable. I have a lot of customers doing 10Gb NFS (I do a lot of VMware) but for true high performance the choice is still Fibre Channel and those same customers are the ones looking heavily at FCoE
I will admit though, performance is impressive.... (Score:4, Interesting)
With no tuning (other than Jumbo frames for FCoE) I was able to get 9.7Gb/s using FCoE over 10Gb ethernet.
While 16Gb FCP/FC is around the corner, you will be able to run FCoE over 40Gb and 100Gb ethernet in 2-3 yrs. (at MUCH $$)
Keep in mind however, iSCSI has been around for over 10yrs now. These things take time to grow, mature, attach.
So lets wait a few more years before declaring anything dead or alive =)
And keep in mind, FCoE is not meant to replace FCP/FC, its meant to fix what is keeping iSCSI from doing better.
too $$$ (Score:2)
not only is FCOE pricey, even gigabit ethernet products are too expensive. they've been out for years - the prices should have dropped by now.
GBIC's Still To Expensive (Score:3)
8gig Fibre Channel GBIC for a SAN fabric averages around $150-$200.
10gig network (CNA) GBIC for a more traditional network averages around $1100.
I am building out a new virtual farm now, and much as we tried to go the converged route with 10gig network, the price point simply isn't there yet (technology is still maturing this year as well). You can work around this with copper for very short runs, but the expense comes in per-rack network gear.
This should start to settle in the fall as the standards fall together better.
Re: (Score:1)
Or you could use Twinax to connect your servers if you're using a top of rack (or similar) topology.
Re: (Score:1)
Re: (Score:2)