Accessing Delete contacts

I am working in Outlook 2010 on an exchange server.  I have delegate rights to my manager's contacts.  I can see them, edit/add when I click on Contacts and choose his mailbox.  However, when I compose an email and click the To; field to get to the address book I cannot see or search for contacts or distrib lists in his contact.  I can only see my own.  Can someone tell me how I can view his contacts from here? 

Thanks.