Query Expressions

Query expressions provide a language integrated syntax for queries that is similar to relational and hierarchical query languages such as SQL and XQuery.

query-expression:
from-clause   query-body
from-clause:
from   from-generators
from-generators:
from-generator
from-generators   ,   from-generator
from-generator:
identifier   in   expression
query-body:
from-or-where-clausesopt   orderby-clauseopt   select-or-group-clause   into-clauseopt
from-or-where-clauses:
from-or-where-clause
from-or-where-clauses   from-or-where-clause
from-or-where-clause:
from-clause
where-clause
where-clause:
where   boolean-expression
orderby-clause:
orderby   ordering-clauses
ordering-clauses:
ordering-clause
ordering-clauses   ,   ordering-clause
ordering-clause:
expression    ordering-directionopt
ordering-direction:
ascending
descending
select-or-group-clause:
select-clause
group-clause
select-clause:
select   expression
group-clause:
group   expression   by   expression
into-clause:
into   identifier   query-body

A query-expression is classified as a non-assignment-expression, the definition of which occurs in §26.3.
A query expression begins with a from clause and ends with either a select or group clause. The initial from clause can be followed by zero or more from or where clauses. Each from clause is a generator that introduces an iteration variable ranging over a sequence, and each where clause is a filter that excludes items from the result. The final select or group clause specifies the shape of the result in terms of the iteration variable(s). The select or group clause may be preceded by an orderby clause that specifies an ordering for the result. Finally, an into clause can be used to "splice" queries by treating the results of one query as a generator in a subsequent query.
In a query expression, a from clause with multiple generators is exactly equivalent to multiple consecutive from clauses with a single generator.

Query Expression Translation

The C# 3.0 language does not specify the exact execution semantics of query expressions. Rather, C# 3.0 translates query expressions into invocations of methods that adhere to the query expression pattern. Specifically, query expressions are translated into invocations of methods named Where, Select, SelectMany, OrderBy, OrderByDescending, ThenBy, ThenByDescending, and GroupBy that are expected to have particular signatures and result types, as described in §26.7.2. These methods can be instance methods of the object being queried or extension methods that are external to the object, and they implement the actual execution of the query.
The translation from query expressions to method invocations is a syntactic mapping that occurs before any type binding or overload resolution has been performed. The translation is guaranteed to be syntactically correct, but it is not guaranteed to produce semantically correct C# code. Following translation of query expressions, the resulting method invocations are processed as regular method invocations, and this may in turn uncover errors, for example if the methods do not exist, if arguments have wrong types, or if the methods are generic and type inference fails.
The translation of query expressions is demonstrated through a series of examples in the following. A formal description of the translation rules is provided in a later section.
where clauses
A where clause in a query expression:

from c in customers
where c.City == "London"
select c

translates to an invocation of a Where method with a synthesized lambda expression created by combining the iteration variable identifier and the expression of the where clause:

customers.
Where(c => c.City == "London")

select clauses
The example in the previous section demonstrates how a select clause that selects the innermost iteration variable is erased by the translation to method invocations.
A select clause that selects something other than the innermost iteration variable:

from c in customers
where c.City == "London"
select c.Name

translates to an invocation of a Select method with a synthesized lambda expression:

customers.
Where(c => c.City == "London").
Select(c => c.Name)

group clauses
A group clause:

from c in customers
group c.Name by c.Country

translates to an invocation of a GroupBy method:

customers.
GroupBy(c => c.Country, c => c.Name)

orderby clauses
An orderby clause:

from c in customers
orderby c.Name
select new { c.Name, c.Phone }

translates to an invocation of an OrderBy method, or an OrderByDescending method if a descending direction was specified:

customers.
OrderBy(c => c.Name).
Select(c => new { c.Name, c.Phone })

Secondary orderings in an orderby clause:

from c in customers
orderby c.Country, c.Balance descending
select new { c.Name, c.Country, c.Balance }

translate to invocations of ThenBy and ThenByDescending methods:

customers.
OrderBy(c => c.Country).
ThenByDescending(c => c.Balance).
Select(c => new { c.Name, c.Country, c.Balance })

Multiple generators
Multiple generators:

from c in customers
where c.City == "London"
from o in c.Orders
where o.OrderDate.Year == 2005
select new { c.Name, o.OrderID, o.Total }

translate to invocations of SelectMany for all but the innermost generator:

customers.
Where(c => c.City == "London").
SelectMany(c =>
   c.Orders.
   Where(o => o.OrderDate.Year == 2005).
   Select(o => new { c.Name, o.OrderID, o.Total })
)

When multiple generators are combined with an orderby clause:

from c in customers, o in c.Orders
where o.OrderDate.Year == 2005
orderby o.Total descending
select new { c.Name, o.OrderID, o.Total }

an additional Select is injected to collect the ordering expressions and the final result in a sequence of tuples. This is necessary such that OrderBy can operate on the entire sequence. Following OrderBy, the final result is extracted from the tuples:

customers.
SelectMany(c =>
   c.Orders.
   Where(o => o.OrderDate.Year == 2005).
   Select(o => new { k1 = o.Total, v = new { c.Name, o.OrderID, o.Total } })
).
OrderByDescending(x => x.k1).
Select(x => x.v)

into clauses
An into clause:

from c in customers
group c by c.Country into g
select new { Country = g.Key, CustCount = g.Group.Count() }

is simply a more convenient notation for a nested query:

from g in
   from c in customers
   group c by c.Country
select new { Country = g.Key, CustCount = g.Group.Count() }

the translation of which is:

customers.
GroupBy(c => c.Country).
Select(g => new { Country = g.Key, CustCount = g.Group.Count() })

The Query Expression Pattern

The Query Expression Pattern establishes a pattern of methods that types can implement to support query expressions. Because query expressions are translated to method invocations by means of a syntactic mapping, types have considerable flexibility in how they implement the query expression pattern. For example, the methods of the pattern can be implemented as instance methods or as extension methods because the two have the same invocation syntax, and the methods can request delegates or expression trees because lambda expressions are convertible to both.
The recommended shape of a generic type C that supports the query expression pattern is shown below. A generic type is used in order to illustrate the proper relationships between parameter and result types, but it is possible to implement the pattern for non-generic types as well.

delegate R Func(A arg);
class C
{
   public C Where(Func predicate);
   public C Select(Func selector);
   public C SelectMany(Func> selector);
   public O OrderBy(Func keyExpr);
   public O OrderByDescending(Func keyExpr);
   public C> GroupBy(Func keyExpr);
   public C> GroupBy(Func keyExpr, Func elemExpr);
}
class O : C
{
   public O ThenBy(Func keySelector);
   public O ThenByDescending(Func keySelector);
}
class G
{
   public K Key { get; }
   public C Group { get; }
}

The methods above use a generic delegate type Func R>, but they could equally well have used other delegate or expression tree types with the same relationships in parameter and result types.
Notice the recommended relationship between C and O which ensures that the ThenBy and ThenByDescending methods are available only on the result of an OrderBy or OrderByDescending. Also notice the recommended shape of the result of GroupBy, which is a sequence of groupings that each has a Key and Group property.
The Standard Query Operators (described in a separate specification) provide an implementation of the query operator pattern for any type that implements the System.Collections.Generic.IEnumerable interface.

Formal Translation Rules

A query expression is processed by repeatedly applying the following translations in order. Each translation is applied until there are no more occurrences of the specified pattern.
Note that in the translations that produce invocations of OrderBy and ThenBy, if the corresponding ordering clause specifies a descending direction indicator, an invocation of OrderByDescending or ThenByDescending is produced instead.

• A query that contains an into clause

q1 into x q2

is translated into

from x in ( q1 ) q2

• A from clause with multiple generators

from g1 , g2 , ... gn

is translated into

from g1 from g2 ... from gn

• A from clause immediately followed by a where clause

from x in e where f

is translated into

from x in ( e ) . Where ( x => f )

• A query expression with multiple from clauses, an orderby clause, and a select clause

from x1 in e1 from x2 in e2 ... orderby k1 , k2 ... select v

is translated into

( from x1 in e1 from x2 in e2 ...
select new { k1 = k1 , k2 = k2 ... , v = v } )
. OrderBy ( x => x . k1 ) . ThenBy ( x => x . k2 ) ...
. Select ( x => x . v )

• A query expression with multiple from clauses, an orderby clause, and a group clause

from x1 in e1 from x2 in e2 ... orderby k1 , k2 ... group v by g

is translated into

( from x1 in e1 from x2 in e2 ...
select new { k1 = k1 , k2 = k2 ... , v = v , g = g } )
. OrderBy ( x => x . k1 ) . ThenBy ( x => x . k2 ) ...
. GroupBy ( x => x . g , x => x . v )

• A query expression with multiple from clauses and a select clause

from x in e from x1 in e1 ... select v

is translated into

( e ) . SelectMany ( x => from x1 in e1 ... select v )

• A query expression with multiple from clauses and a group clause

from x in e from x1 in e1 ... group v by g

is translated into

( e ) . SelectMany ( x => from x1 in e1 ... group v by g )

• A query expression with a single from clause, no orderby clause, and a select clause

from x in e select v

is translated into

( e ) . Select ( x => v )

except when v is the identifier x, the translation is simply

( e )

• A query expression with a single from clause, no orderby clause, and a group clause

from x in e group v by g

is translated into

( e ) . GroupBy ( x => g , x => v )

except when v is the identifier x, the translation is

( e ) . GroupBy ( x => g )

• A query expression with a single from clause, an orderby clause, and a select clause

from x in e orderby k1 , k2 ... select v

is translated into

( e ) . OrderBy ( x => k1 ) . ThenBy ( x => k2 ) ... . Select ( x => v )

except when v is the identifier x, the translation is simply

( e ) . OrderBy ( x => k1 ) . ThenBy ( x => k2 ) ...

• A query expression with a single from clause, an orderby clause, and a group clause

from x in e orderby k1 , k2 ... group v by g

• is translated into

( e ) . OrderBy ( x => k1 ) . ThenBy ( x => k2 ) ...
. GroupBy ( x => g , x => v )

• except when v is the identifier x, the translation is

( e ) . OrderBy ( x => k1 ) . ThenBy ( x => k2 ) ... . GroupBy ( x => g )

Expression Trees

Expression trees permit lambda expressions to be represented as data structures instead of executable code. A lambda expression that is convertible to a delegate type D is also convertible to an expression tree of type System.Query.Expression. Whereas the conversion of a lambda expression to a delegate type causes executable code to be generated and referenced by a delegate, conversion to an expression tree type causes code that creates an expression tree instance to be emitted. Expression trees are efficient in-memory data representations of lambda expressions and make the structure of the expression transparent and explicit.
The following example represents a lambda expression both as executable code and as an expression tree. Because a conversion exists to Func, a conversion also exists to Expression>.

Func f = x => x + 1;                  // Code
Expression> e = x => x + 1;      // Data

Following these assignments, the delegate f references a method that returns x + 1, and the expression tree e references a data structure that describes the expression x + 1.