Introduction

Let $\lambda=(\lambda _{0},\dots,\lambda _{l-1})$ be a partition of $n$, sometimes called a `Young diagram $\lambda'$ of weight $n$. Let $H\sb{\lambda}=J(\lambda\sb{0})\times\cdots\times J(\lambda\sb{l-1})\subset GL(n)$ be the associated maximal abelian subgroup with respect to $\lambda$, where $J(m)$ is the Jordan group of size $m$, i.e.,

$$J(m)=\left\{\sum\sp{m-1}\sb{i=0} h\sb{i}\Lambda\sp{i}\,\vert\... ...n{\bf C}\sp{\times}, h\sb{1},\dots,h\sb{m-1}\in{\bf C}\right\},$$

where the $m \times m$ matrix $\Lambda$ is defined as

$$\font\b=cmr10 scaled\magstep4\Lambda=\pmatrix{ 0 & 1 & &\s... ...s & \cr & {}&\ddots & 1\cr \smash{\hbox{\b 0}}& & {} & 0 \cr }$$

We define the biholomorphic map

$$\begin{eqnarray*} \iota:H\sb{\lambda} & \longrightarrow& \prod_i ({\bf C}\sp{\ti... ...ots, h\sp{(l-1)}\sb{0},\dots, h\sp{(l-1)}\sb{\lambda\sb{l-1}-1}) \end{eqnarray*}$$

where $h=(h\sp{(0)},\dots, h\sp{(l-1)})$, $h\sp{(i)}= \sum_{0\le k<\lambda\sb{i}} h\sp{(i)}\sb{k}\Lambda\sp{k}\in J(\lambda _i)$.

Let $\alpha = (\alpha\sp{(0)} ,\dots , \alpha\sp{(l-1)} ), \alpha^{(i)}:= (\alpha\sb{0}^{(i)} ,\dots , \alpha\sb{\lambda _i-1}^{(i)})\quad (0\le i\le l-1)$ be an $n$-tuple of complex numbers satisfying $\sum_{i=0}^{l-1}\alpha_0\sp{(i)}=-r.$ We define the character $\chi(\cdot\;;\;\alpha ) : \tilde{H}\sb{\lambda}\longrightarrow {\bf C}\sp{\times}$ of the universal covering group $\tilde{H}_{\lambda }=\tilde{J}(\lambda _0) \times\cdots\times\tilde{J}(\lambda _{l-1})$ of $H\sb{\lambda}$ by $\chi(h;\alpha )=\prod_{i=0}^{l-1}\chi(h^{(i)};\alpha ^{(i)})$, where

$$\chi(h^{(i)};\alpha ^{(i)})=h_0^{\alpha _0^{(i)}}\exp\left[\s... ...ts, \frac {h^{(i)}_{\lambda _i-1}}{ h\sb{0}^{(i)}}\big)\right]$$

where $\theta\sb{j}$ are defined as the coefficients of the generating series

$$\log(1+x\sb{1}T+x\sb{2}T\sp{2}+\cdots)=\sum\sp{\infty}\sb{j=0}\theta\sb{j}(x\sb{1},\dots,x\sb{j})T\sp{j}.$$

Recall that the hypergeometric function $\Phi(z;\alpha )$ of type $\lambda$ (see [7]) is a function defined by

$$\Phi(z;\alpha)=\int\sb{\Delta}\chi(\iota\sp{-1}(tz);\alpha)\cdot\omega\quad \mbox{for} \quad z\in Z\sb{r,n}$$ (1)

where $Z\sb{r,n}$ is the set of $r\times n$ complex matrices in general position (see [7]) with respect to $\lambda$, $\omega := \sum_{0\le i<r} (-1)\sp{i} t\sb{i} dt\sb{0}\wedge \cdots \wedge dt\sb{i-1}\wedge dt\sb {i+1}\wedge\cdots\wedge dt\sb{r-1}$ and $\Delta$ is a twisted cycle in the $t$-space depending on $z$ and $\alpha$. Note that for $\lambda =(1,\dots,1)$, the hypergeometric functions of type $\lambda$ coincide with the general hypergeometric function defined in [5].

The set $Z_{r,n}$ admits an action of the group $GL(r)\times H_{\lambda }:$

$$\begin{eqnarray*} GL(r)\times Z_{r,n}\times H_{\lambda }&\longrightarrow& Z_{r,n}\\ (g,z,h)\qquad &\longmapsto& gzh, \end{eqnarray*}$$

under which $\Phi$ behaves as

$$\Phi(gz;\alpha) = ({\rm det}\, g)\sp{-1}\Phi(z;\alpha)\qquad g\in GL(r)$$ (2)

$$\Phi(zh\sb{\lambda};\alpha) = \chi(h\sb{\lambda})\Phi(z;\alpha)\qquad h\sb{\lambda}\in H\sb{\lambda}.$$ (3)

Furthermore, the function $\Phi$ admits another symmetry:

$$\Phi(zw\sb{\lambda};\alpha)=\Phi(z;\alpha {}\sp {t}w\sb{\lambda})\qquad w\sb{\lambda}\in W\sb{\lambda},$$ (4)

where $W\sb{\lambda}$ is an analogue of the Weyl group, see [8].

The hypergeometric functions $\Phi$ on $Z\sb{2,4}$ and $Z\sb{2,5}$ for various partitions $\lambda$ of 4 and 5 were investigated in the papers [7], [9] and [8]. It is known that the functions $\Phi$ are generalizations of Gauss', Kummer's, Bessel's, Hermite's, Airy's functions and the classical hypergeometric functions of two variables, i.e., $F\sb{1}, \Phi\sb{1}, \Phi\sb{2}, \Phi\sb{3}, G\sb{2}, \Gamma\sb{1}, \Gamma\sb{2}$ in Horn's list ([3]). In this paper, we study the hypergeometric functions of type $\lambda$ in two variables on the strata of the set $M(3,6)$ of $3\times 6$ complex matrices.We establish a classification of the functions in terms of the orbital decomposition of the set of strata and give some transformation formulae between some systems of differential equations satisfied by the functions $F\sb{2}, F\sb{3}, H\sb{2}, {\bf H}_3, {\bf H}_{11}, \Psi_1, \Xi_2.$

In Section 1 , we introduce a group $W\sb{\lambda^{(\nu)}}:=R_{\lambda^{(\nu)}} \times \kern -4pt \vert P\sb{\lambda^{(\nu)}}$ which is analogous to the (classical) Weyl group and discuss its properties in detail. In Section 2, we consider the action of $P\sb{\lambda^{(\nu)}}$ on the set $S_{\lambda ^{(\nu)}}$ of strata and obtain the orbital decomposition of $S_{\lambda ^{(\nu)}}$. In Section 3, we obtain suitable normal forms of the matrices in the strata relative to the action of $GL(3)\times H\sb{\lambda}$. Then we can reduce the hypergeometric function $\Phi$ into a function of two variables. In Section 4, we give relations between the classical special functions of hypergeometric type, i.e., Appell's $F\sb{2}$, $F\sb{3}$, $H\sb{2}$ and their confluence in Horn's list, and the hypergeometric functions of type $\lambda$ in two variables. In the last section, some transformation formulae for the systems of diffenentical equations are systematically deduced from the symmetries $(\ref{equation:0.2})$- $(\ref{equation:0.4})$ for the functions $\Phi$.

The author expresses his deep gratitude to Prof. Kazuo Okamoto, Prof. Hironobu Kimura and Prof. Katsunori Iwasaki for their valuable suggestions and kind help during the preparation of this paper.