Kirino 的技术档案

Grain用于读取和处理用于训练和评估 JAX 模型的数据

Unknown — Fri, 15 Aug 2025 16:48:00 +0800

Grain介绍</a></h1>

Grain官方Docs</a></p>

用户可以带来任意的 Python 转换。</li>

Grain 被设计为模块化。如果需要，用户可以使用自己的实现轻松覆盖 Grain 组件。</li>

同一管道的多次运行将产生相同的输出。</li>

Grain 的设计使得检查点的大小最小。抢占后，Grain 可以从中断的地方恢复，并产生与从未被抢占相同的输出。</li>

我们在设计 Grain 时小心翼翼地确保其性能良好（请参阅文档的幕后部分</a>。我们还针对多种数据模式（例如文本/音频/图像/视频）对其进行了测试。</li>

Grain 会尽可能减少其依赖项集。例如，它不应该依赖于 TensorFlow。</li> </ul>

Grain安装</a></h1>

pip</span> install</span> grain</span></span></code></pre>导入所有库</a></h1>
from</span> pathlib</span> import</span> Path</span></span>
import</span> grain</span></span>
import</span> grain</span>.</span>python</span> as</span> pygrain</span></span>
import</span> cv2</span></span>
import</span> albumentations</span> as</span> A</span></span>
import</span> pandas</span> as</span> pd</span></span></code></pre>创建Grain数据源</a></h2>
参考Grain官网要求</a>，数据源需要实现两个魔法方法：</p>
class</span> RandomAccessDataSource</span>(</span>Protocol</span>,</span> Generic</span>[</span>T</span>]</span>)</span>:</span></span>
  """</span>Interface for datasources where storage supports efficient random access.</span>"""</span></span>
</span>
  def</span> __len__</span>(</span>self</span>)</span> -></span> int</span>:</span></span>
    """</span>Number of records in the dataset.</span>"""</span></span>
</span>
  def</span> __getitem__</span>(</span>self</span>,</span> record_key</span>:</span> SupportsIndex</span>)</span> -></span> T</span>:</span></span>
    """</span>Retrieves record for the given record_key.</span>"""</span></span></code></pre>自用数据分类的数据源</a></h3>
from</span> pathlib</span> import</span> Path</span></span>
</span>
</span>
def</span> get_image_extensions</span>(</span>)</span> -></span> tuple</span>[</span>str</span>,</span> ...</span>]</span>:</span></span>
    """</span>Returns a tuple of common image file extensions.</span></span>
</span>
    This function provides a centralized list of supported image file types,</span></span>
    making it easy to manage and update.</span></span>
</span>
    Returns:</span></span>
        A tuple of strings, where each string is an image file extension</span></span>
        (e.g., ".jpg", ".png").</span></span>
</span>
    """</span></span>
    return</span> (</span>"</span>.jpg</span>"</span>,</span> "</span>.jpeg</span>"</span>,</span> "</span>.png</span>"</span>,</span> "</span>.bmp</span>"</span>,</span> "</span>.tiff</span>"</span>,</span> "</span>.tif</span>"</span>,</span> "</span>.webp</span>"</span>)</span></span>
</span>
</span>
class</span> ImageFolderDataSource</span>:</span></span>
    """</span>A data source class that mimics the structure of torchvision.datasets.ImageFolder.</span></span>
</span>
    It expects the root directory to contain subdirectories, where each subdirectory</span></span>
    represents a class and contains images belonging to that class.</span></span>
</span>
    Example directory structure:</span></span>
    root_dir/</span></span>
    ├── class_a/</span></span>
    │   ├── image1.jpg</span></span>
    │   └── image2.png</span></span>
    └── class_b/</span></span>
        ├── image3.jpeg</span></span>
        └── image4.bmp</span></span>
    """</span></span>
</span>
    def</span> __init__</span>(</span>self</span>,</span> root_dir</span>:</span> str</span>)</span>:</span></span>
        """</span>Initializes the ImageFolderDataSource.</span></span>
</span>
        Args:</span></span>
            root_dir: The path to the root directory containing class subdirectories.</span></span>
</span>
        """</span></span>
        self</span>.</span>root_dir</span> =</span> Path</span>(</span>root_dir</span>)</span></span>
        self</span>.</span>samples</span>:</span> list</span>[</span>tuple</span>[</span>str</span>,</span> int</span>]</span>]</span> =</span> [</span>]</span></span>
        self</span>.</span>classes</span>:</span> list</span>[</span>str</span>]</span> =</span> [</span>]</span></span>
        self</span>.</span>_load_samples</span>(</span>)</span></span>
</span>
    def</span> _load_samples</span>(</span>self</span>)</span> -></span> None</span>:</span></span>
        """</span>Loads image file paths and their corresponding class indices from the root directory.</span></span>
</span>
        This method iterates through subdirectories, treats each subdirectory name as a class,</span></span>
        and collects all valid image files within them.</span></span>
        """</span></span>
        if</span> not</span> self</span>.</span>root_dir</span>.</span>exists</span>(</span>)</span>:</span></span>
            msg</span> =</span> f</span>"</span>Root directory </span>{</span>self</span>.</span>root_dir</span>}</span> does not exist.</span>"</span></span>
            raise</span> FileNotFoundError</span>(</span>msg</span>)</span></span>
</span>
        class_to_idx</span> =</span> {</span>}</span></span>
        valid_extensions</span> =</span> get_image_extensions</span>(</span>)</span></span>
</span>
        class_dirs</span> =</span> [</span>d</span> for</span> d</span> in</span> self</span>.</span>root_dir</span>.</span>iterdir</span>(</span>)</span> if</span> d</span>.</span>is_dir</span>(</span>)</span>]</span></span>
        class_dirs</span>.</span>sort</span>(</span>key</span>=</span>lambda</span> x</span>:</span> x</span>.</span>name</span>)</span></span>
</span>
        for</span> class_dir</span> in</span> class_dirs</span>:</span></span>
            class_name</span> =</span> class_dir</span>.</span>name</span></span>
            class_idx</span> =</span> len</span>(</span>class_to_idx</span>)</span></span>
            class_to_idx</span>[</span>class_name</span>]</span> =</span> class_idx</span></span>
            self</span>.</span>classes</span>.</span>append</span>(</span>class_name</span>)</span></span>
</span>
            for</span> ext</span> in</span> valid_extensions</span>:</span></span>
                for</span> img_path</span> in</span> class_dir</span>.</span>glob</span>(</span>f</span>"</span>*</span>{</span>ext</span>}</span>"</span>)</span>:</span></span>
                    self</span>.</span>samples</span>.</span>append</span>(</span>(</span>str</span>(</span>img_path</span>)</span>,</span> class_idx</span>)</span>)</span></span>
</span>
        if</span> not</span> self</span>.</span>samples</span>:</span></span>
            msg</span> =</span> f</span>"</span>No valid images found in directory </span>'</span>{</span>self</span>.</span>root_dir</span>}</span>'</span>"</span></span>
            raise</span> RuntimeError</span>(</span>msg</span>)</span></span>
	#</span> 实现魔法方法__len__</span></span>
    def</span> __len__</span>(</span>self</span>)</span> -></span> int</span>:</span></span>
        """</span>Returns the total number of samples (images) in the dataset.</span></span>
        This allows the dataset object to be used with `len()`.</span></span>
        """</span></span>
        return</span> len</span>(</span>self</span>.</span>samples</span>)</span></span>
	#</span> 实现魔法方法__getitem__</span></span>
    def</span> __getitem__</span>(</span>self</span>,</span> index</span>:</span> int</span>)</span> -></span> tuple</span>[</span>str</span>,</span> int</span>]</span>:</span></span>
        """</span>Returns a sample (image path and its class index) at the given index.</span></span>
        This allows the dataset object to be indexed like a list (e.g., `dataset[0]`).</span></span>
        """</span></span>
        return</span> self</span>.</span>samples</span>[</span>index</span>]</span></span></code></pre>CLIP数据加载DataSource</a></h3>
class</span> CLIPDataSource</span>:</span></span>
    def</span> __init__</span>(</span>self</span>,</span> csv_file</span>)</span>:</span></span>
        df</span> =</span> pd</span>.</span>read_csv</span>(</span>csv_file</span>)</span></span>
        img_paths</span> =</span> df</span>[</span>"</span>image_path</span>"</span>]</span>.</span>tolist</span>(</span>)</span></span>
		texts</span> =</span> df</span>[</span>'</span>txt</span>'</span>]</span>.</span>tolist</span>(</span>)</span></span>
</span>
    def</span> __len__</span>(</span>self</span>)</span> -></span> int</span>:</span></span>
        return</span> len</span>(</span>self</span>.</span>samples</span>)</span></span>
</span>
    def</span> __getitem__</span>(</span>self</span>,</span> idx</span>:</span> int</span>)</span> -></span> tuple</span>[</span>str</span>,</span> str</span>]</span>:</span></span>
        return</span> self</span>.</span>samples</span>[</span>idx</span>]</span></span></code></pre>创建IndexSampler（类似PyTorch的Sampler）</a></h2>
#</span> 实例化数据源</span></span>
train_dataset</span> =</span> ImageFolderDataSource</span>(</span>params</span>.</span>train_data_path</span>)</span></span>
train_sampler</span>=</span>pygrain</span>.</span>IndexSampler</span>(</span></span>
        num_records</span>=</span>len</span>(</span>train_dataset</span>)</span>,</span></span>
        shuffle</span>=</span>True</span>,</span></span>
        seed</span>=</span>params</span>.</span>seed</span>,</span></span>
        shard_options</span>=</span>pygrain</span>.</span>NoSharding</span>(</span>)</span>,</span></span>
        num_epochs</span>=</span>1</span>,</span></span>
    )</span></span></code></pre>创建DataLoader</a></h2>
在grain中需要自己写数据加载流。</p>
OpenCVLoadImageMap</a></h3>
使用OpenCV读取图像</p>
class</span> OpenCVLoadImageMap</span>(</span>grain</span>.</span>transforms</span>.</span>Map</span>)</span>:</span></span>
    def</span> map</span>(</span>self</span>,</span> element</span>:</span> tuple</span>[</span>str</span>,</span> int</span>]</span>)</span> -></span> tuple</span>[</span>np</span>.</span>ndarray</span>,</span> int</span>]</span>:</span></span>
        img_path</span>,</span> label</span> =</span> element</span></span>
        img</span> =</span> cv2</span>.</span>imread</span>(</span>img_path</span>,</span> cv2</span>.</span>IMREAD_COLOR_RGB</span>)</span></span>
        return</span> img</span>,</span> label</span></span></code></pre>CLIPOpenCVLoadImageMap</a></h3>
加载CLIP的图像和文本数据集</p>
class</span> OpenCVLoadImageMap</span>(</span>grain</span>.</span>transforms</span>.</span>Map</span>)</span>:</span></span>
    def</span> map</span>(</span>self</span>,</span> element</span>:</span> tuple</span>[</span>str</span>,</span> str</span>]</span>)</span> -></span> tuple</span>[</span>np</span>.</span>ndarray</span>,</span> str</span>]</span>:</span></span>
        img_path</span>,</span> text</span> =</span> element</span></span>
        img</span> =</span> cv2</span>.</span>imread</span>(</span>img_path</span>,</span> cv2</span>.</span>IMREAD_COLOR_RGB</span>)</span></span>
        return</span> img</span>,</span> text</span></span></code></pre>PILoadImageMap</a></h3>
使用PIL读取图像</p>
class</span> PILoadImageMap</span>(</span>grain</span>.</span>transforms</span>.</span>Map</span>)</span>:</span></span>
    def</span> map</span>(</span>self</span>,</span> element</span>:</span> tuple</span>[</span>str</span>,</span> int</span>]</span>)</span> -></span> tuple</span>[</span>np</span>.</span>ndarray</span>,</span> int</span>]</span>:</span></span>
        img_path</span>,</span> label</span> =</span> element</span></span>
        img</span> =</span> np</span>.</span>asarray</span>(</span>Image</span>.</span>open</span>(</span>img_path</span>)</span>.</span>convert</span>(</span>mode</span>=</span>"</span>RGB</span>"</span>)</span>)</span></span>
        return</span> img</span>,</span> label</span></span></code></pre>AlbumentationsTransform</a></h3>
使用Albumentations进行图像数据增强</p>
class</span> AlbumentationsTransform</span>(</span>grain</span>.</span>transforms</span>.</span>Map</span>)</span>:</span></span>
    def</span> __init__</span>(</span>self</span>,</span> transforms</span>)</span>:</span></span>
        self</span>.</span>transforms</span> =</span> transforms</span></span>
</span>
    def</span> map</span>(</span>self</span>,</span> element</span>:</span> tuple</span>[</span>np</span>.</span>ndarray</span>,</span> int</span>]</span>)</span> -></span> tuple</span>[</span>np</span>.</span>ndarray</span>,</span> int</span>]</span>:</span></span>
        image</span>,</span> label</span> =</span> element</span></span>
        transformed_image</span> =</span> self</span>.</span>transforms</span>(</span>image</span>=</span>image</span>)</span>[</span>"</span>image</span>"</span>]</span></span>
        return</span> transformed_image</span>,</span> label</span></span></code></pre>CLIPAlbumentationsTransform</a></h3>
class</span> CLIPAlbumentationsTransform</span>(</span>grain</span>.</span>transforms</span>.</span>Map</span>)</span>:</span></span>
    def</span> __init__</span>(</span>self</span>,</span> transforms</span>)</span>:</span></span>
        self</span>.</span>transforms</span> =</span> transforms</span></span>
</span>
    def</span> map</span>(</span>self</span>,</span> element</span>:</span> tuple</span>[</span>np</span>.</span>ndarray</span>,</span> str</span>]</span>)</span> -></span> tuple</span>[</span>np</span>.</span>ndarray</span>,</span> str</span>]</span>:</span></span>
        image</span>,</span> text</span> =</span> element</span></span>
        transformed_image</span> =</span> self</span>.</span>transforms</span>(</span>image</span>=</span>image</span>)</span>[</span>"</span>image</span>"</span>]</span></span>
        return</span> transformed_image</span>,</span> text</span></span></code></pre>TokenizerMap</a></h3>
对数据集中的Text进行分词。</p>
class</span> TokenizerMap</span>(</span>grain</span>.</span>transforms</span>.</span>Map</span>)</span>:</span></span>
    def</span> __init__</span>(</span>self</span>,</span> tokenizer</span>,</span> context_length</span>=</span>77</span>)</span>:</span></span>
        self</span>.</span>tokenizer</span> =</span> partial</span>(</span>tokenizer</span>,</span> context_length</span>=</span>context_length</span>)</span></span>
</span>
    def</span> map</span>(</span>self</span>,</span> element</span>:</span> tuple</span>[</span>np</span>.</span>ndarray</span>,</span> str</span>]</span>)</span> -></span> tuple</span>[</span>np</span>.</span>ndarray</span>,</span> np</span>.</span>ndarray</span>]</span>:</span></span>
        img</span>,</span> txt</span> =</span> element</span></span>
        text</span> =</span> self</span>.</span>tokenizer</span>(</span>txt</span>)</span>[</span>0</span>]</span></span>
        return</span> img</span>,</span> text</span></span></code></pre>create_transforms</a></h3>
创建图像增强</p>
def</span> create_transforms</span>(</span>target_size</span>,</span> *</span>, </span>is_training</span>=</span>True</span>)</span> -></span> A</span>.</span>Compose</span>:</span></span>
    """</span>Create image augmentation and normalization transformations.</span></span>
</span>
    Args:</span></span>
        target_size: The desired height and width for the images.</span></span>
        is_training: A boolean indicating whether to apply training-specific augmentations.</span></span>
</span>
    Returns:</span></span>
        An `A.Compose` object containing the sequence of transformations.</span></span>
</span>
    """</span></span>
    transforms_list</span> =</span> [</span></span>
        A</span>.</span>Resize</span>(</span>height</span>=</span>target_size</span>,</span> width</span>=</span>target_size</span>,</span> p</span>=</span>1.0</span>)</span>,</span></span>
    ]</span></span>
</span>
    if</span> is_training</span>:</span></span>
        transforms_list</span>.</span>extend</span>(</span></span>
            [</span></span>
                A</span>.</span>HorizontalFlip</span>(</span>p</span>=</span>0.5</span>)</span>,</span></span>
                A</span>.</span>VerticalFlip</span>(</span>p</span>=</span>0.5</span>)</span>,</span></span>
                A</span>.</span>Rotate</span>(</span>limit</span>=</span>30</span>,</span> p</span>=</span>0.5</span>)</span>,</span></span>
                A</span>.</span>ColorJitter</span>(</span></span>
                    brightness</span>=</span>0.2</span>,</span></span>
                    contrast</span>=</span>0.2</span>,</span></span>
                    saturation</span>=</span>0.2</span>,</span></span>
                    hue</span>=</span>0.05</span>,</span></span>
                    p</span>=</span>0.5</span>,</span></span>
                )</span>,</span></span>
                A</span>.</span>RandomResizedCrop</span>(</span></span>
                    size</span>=</span>(</span>target_size</span>,</span> target_size</span>)</span>,</span></span>
                    scale</span>=</span>(</span>0.8</span>,</span> 1.0</span>)</span>,</span></span>
                    ratio</span>=</span>(</span>0.75</span>,</span> 1.33</span>)</span>,</span></span>
                    p</span>=</span>0.5</span>,</span></span>
                )</span>,</span></span>
            ]</span>,</span></span>
        )</span></span>
</span>
    transforms_list</span>.</span>append</span>(</span></span>
        A</span>.</span>Normalize</span>(</span></span>
            mean</span>=</span>[</span>0.485</span>,</span> 0.456</span>,</span> 0.406</span>]</span>,</span></span>
            std</span>=</span>[</span>0.229</span>,</span> 0.224</span>,</span> 0.225</span>]</span>,</span></span>
            max_pixel_value</span>=</span>255.0</span>,</span></span>
        )</span>,</span></span>
    )</span></span>
</span>
    return</span> A</span>.</span>Compose</span>(</span>transforms_list</span>)</span></span></code></pre>train_loader</span> =</span> pygrain</span>.</span>DataLoader</span>(</span></span>
        data_source</span>=</span>train_dataset</span>,</span></span>
        sampler</span>=</span>train_sampler</span>,</span></span>
        worker_count</span>=</span>params</span>.</span>num_workers</span>,</span></span>
        worker_buffer_size</span>=</span>2</span>,</span></span>
        operations</span>=</span>[</span></span>
            OpenCVLoadImageMap</span>(</span>)</span>,</span></span>
            AlbumentationsTransform</span>(</span></span>
                create_transforms</span>(</span></span>
                    target_size</span>=</span>params</span>.</span>target_size</span>,</span></span>
                    is_training</span>=</span>True</span>,</span></span>
                )</span>,</span></span>
            )</span>,</span></span>
            pygrain</span>.</span>Batch</span>(</span></span>
                params</span>.</span>batch_size</span>,</span></span>
                drop_remainder</span>=</span>True</span>,</span></span>
            )</span>,</span></span>
        ]</span>,</span></span>
    )</span></span></code></pre>


LeNet（Jax/Flax）实现
Unknown — Fri, 15 Aug 2025 16:23:00 +0800
LeNet</a></h1>
LeNet，它是最早发布的卷积神经网络之一，因其在计算机视觉任务中的高效性能而受到广泛关注。 这个模型是由AT&T贝尔实验室的研究员Yann LeCun在1989年提出的（并以其命名），目的是识别图像 (LeCun et al., 1998)中的手写数字。 当时，Yann LeCun发表了第一篇通过反向传播成功训练卷积神经网络的研究，这项工作代表了十多年来神经网络研究开发的成果。
当时，LeNet取得了与支持向量机（support vector machines）性能相媲美的成果，成为监督学习的主流方法。 LeNet被广泛用于自动取款机（ATM）机中，帮助识别处理支票的数字。 时至今日，一些自动取款机仍在运行Yann LeCun和他的同事Leon Bottou在上世纪90年代写的代码。</p>
模型结构</a></h1>

总体来看，LeNet（LeNet-5）由两个部分组成：</p>

卷积编码器：由两个卷积层组成;</li>
全连接层密集块：由三个全连接层组成。</li>
</ul>
Flax实现</a></h1>
import</span> jax</span></span>
import</span> jax</span>.</span>numpy</span> as</span> jnp</span></span>
from</span> flax</span> import</span> nnx</span></span>
from</span> functools</span> import</span> partial</span></span>
</span>
</span>
class</span> LeNet</span>(</span>nnx</span>.</span>Module</span>)</span>:</span></span>
    def</span> __init__</span>(</span>self</span>,</span> *</span>, </span>rngs</span>:</span> nnx</span>.</span>Rngs</span>)</span>:</span></span>
        self</span>.</span>features</span> =</span> nnx</span>.</span>Sequential</span>(</span></span>
            nnx</span>.</span>Conv</span>(</span></span>
                in_features</span>=</span>1</span>,</span></span>
                out_features</span>=</span>6</span>,</span></span>
                kernel_size</span>=</span>(</span>5</span>,</span> 5</span>)</span>,</span></span>
                padding</span>=</span>(</span>2</span>,</span> 2</span>)</span>,</span></span>
                rngs</span>=</span>rngs</span>,</span></span>
            )</span>,</span></span>
            nnx</span>.</span>sigmoid</span>,</span></span>
            partial</span>(</span>nnx</span>.</span>avg_pool</span>,</span> window_shape</span>=</span>(</span>2</span>,</span> 2</span>)</span>,</span> strides</span>=</span>(</span>2</span>,</span> 2</span>)</span>)</span>,</span></span>
            nnx</span>.</span>Conv</span>(</span></span>
                in_features</span>=</span>6</span>,</span></span>
                out_features</span>=</span>16</span>,</span></span>
                kernel_size</span>=</span>(</span>5</span>,</span> 5</span>)</span>,</span></span>
                padding</span>=</span>"</span>VALID</span>"</span>,</span></span>
                rngs</span>=</span>rngs</span>,</span></span>
            )</span>,</span></span>
            nnx</span>.</span>sigmoid</span>,</span></span>
            partial</span>(</span>nnx</span>.</span>avg_pool</span>,</span> window_shape</span>=</span>(</span>2</span>,</span> 2</span>)</span>,</span> strides</span>=</span>(</span>2</span>,</span> 2</span>)</span>)</span>,</span></span>
        )</span></span>
        self</span>.</span>fc</span> =</span> nnx</span>.</span>Sequential</span>(</span></span>
            lambda</span> x</span>:</span> x</span>.</span>reshape</span>(</span>x</span>.</span>shape</span>[</span>0</span>]</span>,</span> -</span>1</span>)</span>,</span></span>
            nnx</span>.</span>Linear</span>(</span>in_features</span>=</span>16</span> *</span> 5</span> *</span> 5</span>,</span> out_features</span>=</span>120</span>,</span> rngs</span>=</span>rngs</span>)</span>,</span></span>
            nnx</span>.</span>sigmoid</span>,</span></span>
            nnx</span>.</span>Linear</span>(</span>in_features</span>=</span>120</span>,</span> out_features</span>=</span>84</span>,</span> rngs</span>=</span>rngs</span>)</span>,</span></span>
            nnx</span>.</span>sigmoid</span>,</span></span>
            nnx</span>.</span>Linear</span>(</span>in_features</span>=</span>84</span>,</span> out_features</span>=</span>10</span>,</span> rngs</span>=</span>rngs</span>)</span>,</span></span>
        )</span></span>
</span>
    def</span> __call__</span>(</span>self</span>,</span> x</span>)</span>:</span></span>
        x</span> =</span> self</span>.</span>features</span>(</span>x</span>)</span></span>
        return</span> self</span>.</span>fc</span>(</span>x</span>)</span></span></code></pre>模型测试</a></h1>
rngs</span> =</span> nnx</span>.</span>Rngs</span>(</span>42</span>)</span></span>
net</span> =</span> LeNet</span>(</span>rngs</span>=</span>rngs</span>)</span></span>
X</span> =</span> jax</span>.</span>random</span>.</span>normal</span>(</span>rngs</span>.</span>params</span>(</span>)</span>,</span> shape</span>=</span>(</span>1</span>,</span> 28</span>,</span> 28</span>,</span> 1</span>)</span>,</span> dtype</span>=</span>jnp</span>.</span>float32</span>)</span></span>
</span>
#</span> Features</span></span>
for</span> i</span>,</span> layer</span> in</span> enumerate</span>(</span>net</span>.</span>features</span>.</span>layers</span>)</span>:</span></span>
    X</span> =</span> layer</span>(</span>X</span>)</span></span>
    print</span>(</span>f</span>"</span>features[</span>{</span>i</span>}</span>] (</span>{</span>layer</span>.</span>__class__</span>.</span>__name__</span>}</span>): 输出形状 </span>{</span>X</span>.</span>shape</span>}</span>"</span>)</span></span>
#</span> Fc</span></span>
for</span> i</span>,</span> layer</span> in</span> enumerate</span>(</span>net</span>.</span>fc</span>.</span>layers</span>)</span>:</span></span>
    X</span> =</span> layer</span>(</span>X</span>)</span></span>
    print</span>(</span>f</span>"</span>fc[</span>{</span>i</span>}</span>] (</span>{</span>layer</span>.</span>__class__</span>.</span>__name__</span>}</span>): 输出形状 </span>{</span>X</span>.</span>shape</span>}</span>"</span>)</span></span></code></pre>
输出</p>
features[0]</span> (Conv</span>): 输出形状 (</span>1,</span> 28,</span> 28,</span> 6</span>)</span></span>
features[1]</span> (PjitFunction</span>): 输出形状 (</span>1,</span> 28,</span> 28,</span> 6</span>)</span></span>
features[2]</span> (partial</span>): 输出形状 (</span>1,</span> 14,</span> 14,</span> 6</span>)</span></span>
features[3]</span> (Conv</span>): 输出形状 (</span>1,</span> 10,</span> 10,</span> 16</span>)</span></span>
features[4]</span> (PjitFunction</span>): 输出形状 (</span>1,</span> 10,</span> 10,</span> 16</span>)</span></span>
features[5]</span> (partial</span>): 输出形状 (</span>1,</span> 5,</span> 5,</span> 16</span>)</span></span>
fc</span>[</span>0</span>]</span> (function</span>): 输出形状 (</span>1,</span> 400</span>)</span></span>
fc</span>[</span>1</span>]</span> (Linear</span>): 输出形状 (</span>1,</span> 120</span>)</span></span>
fc</span>[</span>2</span>]</span> (PjitFunction</span>): 输出形状 (</span>1,</span> 120</span>)</span></span>
fc</span>[</span>3</span>]</span> (Linear</span>): 输出形状 (</span>1,</span> 84</span>)</span></span>
fc</span>[</span>4</span>]</span> (PjitFunction</span>): 输出形状 (</span>1,</span> 84</span>)</span></span>
fc</span>[</span>5</span>]</span> (Linear</span>): 输出形状 (</span>1,</span> 10</span>)</span></span></code></pre>


LeNet（PyTorch）实现
Unknown — Fri, 15 Aug 2025 15:59:00 +0800
LeNet</a></h1>
LeNet，它是最早发布的卷积神经网络之一，因其在计算机视觉任务中的高效性能而受到广泛关注。 这个模型是由AT&T贝尔实验室的研究员Yann LeCun在1989年提出的（并以其命名），目的是识别图像 (LeCun et al., 1998)中的手写数字。 当时，Yann LeCun发表了第一篇通过反向传播成功训练卷积神经网络的研究，这项工作代表了十多年来神经网络研究开发的成果。
当时，LeNet取得了与支持向量机（support vector machines）性能相媲美的成果，成为监督学习的主流方法。 LeNet被广泛用于自动取款机（ATM）机中，帮助识别处理支票的数字。 时至今日，一些自动取款机仍在运行Yann LeCun和他的同事Leon Bottou在上世纪90年代写的代码。</p>
模型结构</a></h1>

总体来看，LeNet（LeNet-5）由两个部分组成：</p>

卷积编码器：由两个卷积层组成;</li>
全连接层密集块：由三个全连接层组成。</li>
</ul>
PyTorch实现</a></h1>
import</span> torch</span></span>
import</span> torch</span>.</span>nn</span> as</span> nn</span></span>
</span>
</span>
class</span> LeNet</span>(</span>nn</span>.</span>Module</span>)</span>:</span></span>
    def</span> __init__</span>(</span>self</span>)</span>:</span></span>
        super</span>(</span>LeNet</span>,</span> self</span>)</span>.</span>__init__</span>(</span>)</span></span>
        self</span>.</span>features</span> =</span> nn</span>.</span>Sequential</span>(</span></span>
            nn</span>.</span>Conv2d</span>(</span>in_channels</span>=</span>1</span>,</span> out_channels</span>=</span>6</span>,</span> kernel_size</span>=</span>5</span>,</span> padding</span>=</span>2</span>)</span>,</span></span>
            nn</span>.</span>Sigmoid</span>(</span>)</span>,</span></span>
            nn</span>.</span>AvgPool2d</span>(</span>kernel_size</span>=</span>2</span>,</span> stride</span>=</span>2</span>)</span>,</span></span>
            nn</span>.</span>Conv2d</span>(</span>in_channels</span>=</span>6</span>,</span> out_channels</span>=</span>16</span>,</span> kernel_size</span>=</span>5</span>)</span>,</span></span>
            nn</span>.</span>Sigmoid</span>(</span>)</span>,</span></span>
            nn</span>.</span>AvgPool2d</span>(</span>kernel_size</span>=</span>2</span>,</span> stride</span>=</span>2</span>)</span>,</span></span>
        )</span></span>
        self</span>.</span>fc</span> =</span> nn</span>.</span>Sequential</span>(</span></span>
            nn</span>.</span>Flatten</span>(</span>)</span>,</span></span>
            nn</span>.</span>Linear</span>(</span>in_features</span>=</span>16</span> *</span> 5</span> *</span> 5</span>,</span> out_features</span>=</span>120</span>)</span>,</span></span>
            nn</span>.</span>Sigmoid</span>(</span>)</span>,</span></span>
            nn</span>.</span>Linear</span>(</span>in_features</span>=</span>120</span>,</span> out_features</span>=</span>84</span>)</span>,</span></span>
            nn</span>.</span>Sigmoid</span>(</span>)</span>,</span></span>
            nn</span>.</span>Linear</span>(</span>in_features</span>=</span>84</span>,</span> out_features</span>=</span>10</span>)</span>,</span></span>
        )</span></span>
</span>
    def</span> forward</span>(</span>self</span>,</span> x</span>)</span>:</span></span>
        x</span> =</span> self</span>.</span>features</span>(</span>x</span>)</span></span>
        return</span> self</span>.</span>fc</span>(</span>x</span>)</span></span></code></pre>模型测试</a></h1>
net</span> =</span> LeNet</span>(</span>)</span></span>
X</span> =</span> torch</span>.</span>rand</span>(</span>size</span>=</span>(</span>1</span>,</span> 1</span>,</span> 28</span>,</span> 28</span>)</span>,</span> dtype</span>=</span>torch</span>.</span>float32</span>)</span></span>
print</span>(</span>f</span>"</span>Initial input shape: </span>{</span>X</span>.</span>shape</span>}</span>\n</span>"</span>)</span></span>
X_features</span> =</span> X</span></span>
print</span>(</span>"</span>--- Features Block ---</span>"</span>)</span></span>
for</span> i</span>,</span> layer</span> in</span> enumerate</span>(</span>net</span>.</span>features</span>)</span>:</span></span>
    X_features</span> =</span> layer</span>(</span>X_features</span>)</span></span>
    print</span>(</span></span>
        f</span>"</span>Features[</span>{</span>i</span>}</span>] (</span>{</span>layer</span>.</span>__class__</span>.</span>__name__</span>}</span>): output shape </span>{</span>X_features</span>.</span>shape</span>}</span>"</span></span>
    )</span></span>
</span>
print</span>(</span>"</span>\n</span>--- Fully Connected Block ---</span>"</span>)</span></span>
X_fc</span> =</span> X_features</span></span>
for</span> i</span>,</span> layer</span> in</span> enumerate</span>(</span>net</span>.</span>fc</span>)</span>:</span></span>
    X_fc</span> =</span> layer</span>(</span>X_fc</span>)</span></span>
    print</span>(</span>f</span>"</span>FC[</span>{</span>i</span>}</span>] (</span>{</span>layer</span>.</span>__class__</span>.</span>__name__</span>}</span>): output shape </span>{</span>X_fc</span>.</span>shape</span>}</span>"</span>)</span></span></code></pre>
输出</p>
Initial</span> input</span> shape:</span> torch.Size</span>(</span>[</span>1, 1, 28, 28</span>]</span>)</span></span>
</span>
---</span> Features</span> Block</span> -</span>--</span></span>
Features[0]</span> (Conv2d</span>): output shape torch.Size(</span>[</span>1, 6, 28, 28</span>]</span>)</span></span>
Features[1]</span> (Sigmoid</span>): output shape torch.Size(</span>[</span>1, 6, 28, 28</span>]</span>)</span></span>
Features[2]</span> (AvgPool2d</span>): output shape torch.Size(</span>[</span>1, 6, 14, 14</span>]</span>)</span></span>
Features[3]</span> (Conv2d</span>): output shape torch.Size(</span>[</span>1, 16, 10, 10</span>]</span>)</span></span>
Features[4]</span> (Sigmoid</span>): output shape torch.Size(</span>[</span>1, 16, 10, 10</span>]</span>)</span></span>
Features[5]</span> (AvgPool2d</span>): output shape torch.Size(</span>[</span>1, 16, 5, 5</span>]</span>)</span></span>
</span>
---</span> Fully</span> Connected</span> Block</span> -</span>--</span></span>
FC[0]</span> (Flatten</span>): output shape torch.Size(</span>[</span>1, 400</span>]</span>)</span></span>
FC[1]</span> (Linear</span>): output shape torch.Size(</span>[</span>1, 120</span>]</span>)</span></span>
FC[2]</span> (Sigmoid</span>): output shape torch.Size(</span>[</span>1, 120</span>]</span>)</span></span>
FC[3]</span> (Linear</span>): output shape torch.Size(</span>[</span>1, 84</span>]</span>)</span></span>
FC[4]</span> (Sigmoid</span>): output shape torch.Size(</span>[</span>1, 84</span>]</span>)</span></span>
FC[5]</span> (Linear</span>): output shape torch.Size(</span>[</span>1, 10</span>]</span>)</span></span></code></pre>


数据挖掘第6次作业
Unknown — Sat, 22 Apr 2023 10:49:00 +0800
电子商务网站用户行为分析及服务推荐</a></h1>
一、背景与挖掘目标</a></h2>
本章通过对用户访问的网页日志数据进行分析与处理，采用基于物品的协同过滤算法对处理后的数据进行建模分析，并应用模型实现智能推荐，进行个性化推荐，帮助用户更加便捷地获取信息。</p>
某法律网站是北京一家电子商务类的大型法律资讯网站，致力于为用户提供丰富的法律信息与专业咨询服务，本案例主要是为律师与律师事务所提供互联网整合营销解决方案。随着企业经营水平的提高，其网站访问量逐步增加，随之而来的数据信息量也在大幅增长。带来的问题是用户在面对大量信息时无法快速获取需要的信息，使得信息使用效率降低。用户在浏览、搜寻想要的信息的过程中需要花费大量的时间，这种情况的出现造成了用户的不断流失，对企业造成巨大的损失。</p>
为了节省用户时间并帮助用户快速找到感兴趣的信息，利用网站海量的用户访问数据研究用户的兴趣偏好，分析用户的需求和行为，引导用户发现需求信息，将长尾网页准确地推荐给所需用户，帮助用户发现他们感兴趣但很难发现的网页信息。总而言之，智能推荐服务可以为用户提供个性化的服务、改善用户浏览体验、增加用户黏性、从而使用户与企业之间建立稳定的交互关系，实现客户链式反应增值。</p>
二、分析方法与过程</a></h2>
（一）分析步骤与流程</a></h3>
为了帮助用户从海量的信息中快速发现感兴趣的网页，本案例主要采用协同过滤算法进行推荐，其推荐原理如图所示。</p>
由于用户访问网站的数据记录较多，若不进行分类处理直接采用协同过滤算法进行推荐，会存在以下问题:1)数据量大说明物品数与用户数较多，在模型构建用户与物品的稀疏矩阵时，模型计算需要消耗大量的时间，并且会造成设备内存空间不足的问题。2)不同的用户关注的信息不同，其推荐结果不能满足用户的个性化需求。</p>
为了避免上述问题，需要对用户访问记录进行分类处理与分析。在用户访问记录日志中，没有用户访问网页时间长短的记录，不能根据用户在网页的停留时间判断用户是否对浏览网页感兴趣。</p>
本案例采用基于用户浏览网页的类型的方法进行分类，然后对每个类型中的内容进行智能推荐。采用上述分析方法与思路，结合原始数据及分析目标，整理的网站智能推荐流程如下图所示，主要步骤如下:</p>
1)从系统中获取用户访问网站的原始记录。</p>
2)分析用户访问内容、用户流失等。</p>
3)对数据进行预处理，包含数据去重、数据变换等过程。</p>
4)以用户访问html后缀的网页为关键条件，对数据进行处理。</p>
5)对比多种推荐算法的效果，选择效果较好的模型。通过模型预 测，获得推荐结果。</p>
（二）数据抽取</a></h3>
以用户的访问时间为条件，选取3个月内(2015年2月1日至2015年4月29日)用户的访问数据作为原始数据集。由于每个地区的用户访问习惯以及兴趣爱好存在差异性，因此抽取广州地区的用户访问数据进行分析，其数据量总共有837450条，其中包括用户号、访问时间、来源网站、访问页面、页面标题、来源网页、标签、网页类别、关键词等。</p>
在数据抽取过程中，由于数据量较大且存储在数据库中，为了提高数据处理的效率，采取用Python读取数据库的操作方式。本案例用到的数据库为开源数据库(MySQL-community-5.6.39.0)。安装数据库后导入本案例的数据原始文件7law.sql，然后可以利用Python对MySQL数据库进行连接以及其他相关操作，如以下代码。</p>
首先在MySQL中创建test数据库，然后把表数据导入数据库，最后连接数据库并选取3个月内用户的访问数据，如以下代码所示。</p>
1</span> import</span> os</span></span>
 2</span> import</span> pandas</span> as</span> pd</span></span>
 3</span> </span></span>
 4</span> </span></span>
 5</span> #</span> 修改工作路径到指定文件夹</span></span>
 6</span> os</span>.</span>chdir</span>(</span>"</span>E:/大三下/数据分析/数据/第十一章</span>"</span>)</span></span>
 7</span> </span></span>
 8</span> #</span> 第一种连接方式</span></span>
 9</span> #</span>from sqlalchemy import create_engine</span></span>
10</span> </span></span>
11</span> #</span>engine = create_engine('mysql+pymysql://root:123@192.168.31.140:3306/test1?charset=utf8')</span></span>
12</span> #</span>sql = pd.read_sql('all_gzdata', engine, chunksize = 10000)</span></span>
13</span> </span></span>
14</span> #</span> 第二种连接方式</span></span>
15</span> import</span> pymysql</span> as</span> pm</span></span>
16</span> </span></span>
17</span> con</span> =</span> pm</span>.</span>connect</span>(</span>host</span> =</span> '</span>localhost</span>'</span>,</span>user</span> =</span> '</span>root</span>'</span>,</span>password</span> =</span> '</span>123456</span>'</span>,</span>database</span> =</span> '</span>test1</span>'</span>,</span>charset</span>=</span>'</span>utf8</span>'</span>)</span></span>
18</span> data</span> =</span> pd</span>.</span>read_sql</span>(</span>'</span>select * from all_gzdata</span>'</span>,</span>con</span>=</span>con</span>)</span></span>
19</span> con</span>.</span>close</span>(</span>)</span>           #</span>关闭连接</span></span>
20</span> </span></span>
21</span> #</span> 保存读取的数据</span></span>
22</span> data</span>.</span>to_csv</span>(</span>'</span>E:/大三下/数据分析/数据/第十一章/all_gzdata.csv</span>'</span>,</span> index</span>=</span>False</span>,</span> encoding</span>=</span>'</span>utf-8</span>'</span>)</span></span></code></pre>（三）数据探索分析</a></h3>
原始数据集中包括用户号、访问时间、来源网站、访问页面、页面标题、来源网页、标签、网页类别和关键词等信息，需要对原始数据进行网页类型、点击次数、网页排名等各个维度的分布分析，了解用户浏览网页的行为及关注内容，获得数据内在的规律。</p>
1、分析网页类型</a></h4>
对原始数据中用户点击的网页类型进行统计分析，如以下代码所示。</p>
 1</span> import</span> pandas</span> as</span> pd</span></span>
 2</span> from</span> sqlalchemy</span> import</span> create_engine</span></span>
 3</span> </span></span>
 4</span> engine</span> =</span> create_engine</span>(</span>'</span>mysql+pymysql://root:123456@127.0.0.1:3306/test1?charset=utf8</span>'</span>)</span></span>
 5</span> sql</span> =</span> pd</span>.</span>read_sql</span>(</span>'</span>all_gzdata</span>'</span>,</span> engine</span>,</span> chunksize</span> =</span> 10000</span>)</span></span>
 6</span> #</span> 分析网页类型</span></span>
 7</span> counts</span> =</span> [</span>i</span>[</span>'</span>fullURLId</span>'</span>]</span>.</span>value_counts</span>(</span>)</span> for</span> i</span> in</span> sql</span>]</span> #</span>逐块统计</span></span>
 8</span> counts</span> =</span> counts</span>.</span>copy</span>(</span>)</span></span>
 9</span> counts</span> =</span> pd</span>.</span>concat</span>(</span>counts</span>)</span>.</span>groupby</span>(</span>level</span>=</span>0</span>)</span>.</span>sum</span>(</span>)</span>  #</span> 合并统计结果，把相同的统计项合并（即按index分组并求和）</span></span>
10</span> counts</span> =</span> counts</span>.</span>reset_index</span>(</span>)</span>  #</span> 重新设置index，将原来的index作为counts的一列。</span></span>
11</span> counts</span>.</span>columns</span> =</span> [</span>'</span>index</span>'</span>,</span> '</span>num</span>'</span>]</span>  #</span> 重新设置列名，主要是第二列，默认为0</span></span>
12</span> counts</span>[</span>'</span>type</span>'</span>]</span> =</span> counts</span>[</span>'</span>index</span>'</span>]</span>.</span>str</span>.</span>extract</span>(</span>'</span>(\d</span>{3}</span>)</span>'</span>)</span>  #</span> 提取前三个数字作为类别id</span></span>
13</span> counts_</span> =</span> counts</span>[</span>[</span>'</span>type</span>'</span>,</span> '</span>num</span>'</span>]</span>]</span>.</span>groupby</span>(</span>'</span>type</span>'</span>)</span>.</span>sum</span>(</span>)</span>  #</span> 按类别合并</span></span>
14</span> counts_</span>.</span>sort_values</span>(</span>by</span>=</span>'</span>num</span>'</span>,</span> ascending</span>=</span>False</span>,</span> inplace</span>=</span>True</span>)</span>  #</span> 降序排列</span></span>
15</span> counts_</span>[</span>'</span>ratio</span>'</span>]</span> =</span> counts_</span>.</span>iloc</span>[</span>:</span>,</span>0</span>]</span> /</span> counts_</span>.</span>iloc</span>[</span>:</span>,</span>0</span>]</span>.</span>sum</span>(</span>)</span></span>
16</span> print</span>(</span>counts_</span>)</span></span></code></pre>
通过输出结果可以发现，点击与咨询相关(网页类型为101，http://www..com/ask/)的记录占了49.16%，其他类型(网页类型为199)占比24%左右，知识相关(网页类型为107，http://www.</strong></strong>.com/info/)占比22%左右。</p>
根据统计结果对用户点击的页面类型进行排名，依次为咨询相关、知识相关、其他方面的网页、法规(类型为301)、律师相关(类型为102)等。然后进一步对咨询类别内部进行统计分析，如下述代码所示。</p>
 1</span> #</span> 因为只有107001一类，但是可以继续细分成三类：知识内容页、知识列表页、知识首页</span></span>
 2</span> def</span> count107</span>(</span>i</span>)</span>:</span> #</span>自定义统计函数</span></span>
 3</span>     j</span> =</span> i</span>[</span>[</span>'</span>fullURL</span>'</span>]</span>]</span>[</span>i</span>[</span>'</span>fullURLId</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>107</span>'</span>)</span>]</span>.</span>copy</span>(</span>)</span>  #</span> 找出类别包含107的网址</span></span>
 4</span>     j</span>[</span>'</span>type</span>'</span>]</span> =</span> None</span> #</span> 添加空列</span></span>
 5</span>     j</span>[</span>'</span>type</span>'</span>]</span>[</span>j</span>[</span>'</span>fullURL</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>info/.+?/</span>'</span>)</span>]</span>=</span> '</span>知识首页</span>'</span></span>
 6</span>     j</span>[</span>'</span>type</span>'</span>]</span>[</span>j</span>[</span>'</span>fullURL</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>info/.+?/.+?</span>'</span>)</span>]</span>=</span> '</span>知识列表页</span>'</span></span>
 7</span>     j</span>[</span>'</span>type</span>'</span>]</span>[</span>j</span>[</span>'</span>fullURL</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>/\d+?_*\d+?\.html</span>'</span>)</span>]</span>=</span> '</span>知识内容页</span>'</span></span>
 8</span>     return</span> j</span>[</span>'</span>type</span>'</span>]</span>.</span>value_counts</span>(</span>)</span></span>
 9</span> #</span> 注意：获取一次sql对象就需要重新访问一下数据库(!!!)</span></span>
10</span> #</span>engine = create_engine('mysql+pymysql://root:123456@127.0.0.1:3306/test?charset=utf8')</span></span>
11</span> sql</span> =</span> pd</span>.</span>read_sql</span>(</span>'</span>all_gzdata</span>'</span>,</span> engine</span>,</span> chunksize</span> =</span> 10000</span>)</span></span>
12</span> </span></span>
13</span> counts2</span> =</span> [</span>count107</span>(</span>i</span>)</span> for</span> i</span> in</span> sql</span>]</span> #</span> 逐块统计</span></span>
14</span> counts2</span> =</span> pd</span>.</span>concat</span>(</span>counts2</span>)</span>.</span>groupby</span>(</span>level</span>=</span>0</span>)</span>.</span>sum</span>(</span>)</span>  #</span> 合并统计结果</span></span>
15</span> print</span>(</span>counts2</span>)</span></span>
16</span> #</span>计算各个部分的占比</span></span>
17</span> res107</span> =</span> pd</span>.</span>DataFrame</span>(</span>counts2</span>)</span></span>
18</span> #</span> res107.reset_index(inplace=True)</span></span>
19</span> res107</span>.</span>index</span>.</span>name</span>=</span> '</span>107类型</span>'</span></span>
20</span> res107</span>.</span>rename</span>(</span>columns</span>=</span>{</span>'</span>type</span>'</span>:</span>'</span>num</span>'</span>}</span>,</span> inplace</span>=</span>True</span>)</span></span>
21</span> res107</span>[</span>'</span>比例</span>'</span>]</span> =</span> res107</span>[</span>'</span>num</span>'</span>]</span> /</span> res107</span>[</span>'</span>num</span>'</span>]</span>.</span>sum</span>(</span>)</span></span>
22</span> res107</span>.</span>reset_index</span>(</span>inplace</span> =</span> True</span>)</span></span>
23</span> print</span>(</span>res107</span>)</span></span></code></pre>
通过输出结果可以发现，浏览咨询内容页(101003)记录最多，其次是咨询列表页(101002)和咨询首页(101001)。初步分析可以得知用户都喜欢通过浏览问题的方式找到自己需要的信息，而不是以提问的方式或者查看长篇内容的方式寻找信息。</p>
通过分析其他(199)页面的情况可知，其中网址中带有“?”的占了 32%左右，其他咨询相关与法规专题占比达到41%，地区和律师占比为 20%左右。同时，在进行网页分类过程中发现，律师、地区、咨询相关的网页还会在其他类别中存在，并且大部分是以以下网址的形式存在:</p>
1)http://www.****.com/guangzhou/p2lawfirm，地区律师事务所。</p>
2)http://www.****.com/guangzhou，地区网址。</p>
3)http://www.****.com/ask/ask.php，咨询内容提交页。</p>
4)http://www.****.com/ask/midques_10549897.html，中间类型网 页。</p>
5)http://www.****.com/ask/exp/4317.html，咨询经验。</p>
6)http://www.****.com/ask/online/138.html，在线咨询页。</p>
首先是网址中带有lawfirm关键字的对应律师事务所，其次是带有 ask/exp、ask/online关键字的对应咨询经验和在线咨询页。大多数用户浏览网页的情况为咨询内容页、知识内容页、法规专题页、在线咨询页等，其中咨询内容页和知识内容页占比最高。对原始数据的网址中带“?”的数据进行统计，如以下代码所示。</p>
1</span> def</span> countquestion</span>(</span>i</span>)</span>:</span>  #</span> 自定义统计函数</span></span>
 2</span>     j</span> =</span> i</span>[</span>[</span>'</span>fullURLId</span>'</span>]</span>]</span>[</span>i</span>[</span>'</span>fullURL</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>\?</span>'</span>)</span>]</span>.</span>copy</span>(</span>)</span>  #</span> 找出类别包含107的网址</span></span>
 3</span>     return</span> j</span></span>
 4</span> </span></span>
 5</span> #</span>engine = create_engine('mysql+pymysql://root:123456@127.0.0.1:3306/test?charset=utf8')</span></span>
 6</span> sql</span> =</span> pd</span>.</span>read_sql</span>(</span>'</span>all_gzdata</span>'</span>,</span> engine</span>,</span> chunksize</span> =</span> 10000</span>)</span></span>
 7</span> </span></span>
 8</span> counts3</span> =</span> [</span>countquestion</span>(</span>i</span>)</span>[</span>'</span>fullURLId</span>'</span>]</span>.</span>value_counts</span>(</span>)</span> for</span> i</span> in</span> sql</span>]</span></span>
 9</span> counts3</span> =</span> pd</span>.</span>concat</span>(</span>counts3</span>)</span>.</span>groupby</span>(</span>level</span>=</span>0</span>)</span>.</span>sum</span>(</span>)</span></span>
10</span> print</span>(</span>counts3</span>)</span></span>
11</span> </span></span>
12</span> #</span> 求各个类型的占比并保存数据</span></span>
13</span> df1</span> =</span>  pd</span>.</span>DataFrame</span>(</span>counts3</span>)</span></span>
14</span> df1</span>[</span>'</span>perc</span>'</span>]</span> =</span> df1</span>[</span>'</span>fullURLId</span>'</span>]</span>/</span>df1</span>[</span>'</span>fullURLId</span>'</span>]</span>.</span>sum</span>(</span>)</span>*</span>100</span></span>
15</span> df1</span>.</span>sort_values</span>(</span>by</span>=</span>'</span>fullURLId</span>'</span>,</span>ascending</span>=</span>False</span>,</span>inplace</span>=</span>True</span>)</span></span>
16</span> print</span>(</span>df1</span>.</span>round</span>(</span>4</span>)</span>)</span></span></code></pre>
通过输出结果可以看出，网址中带有“?”的记录一共有65492条，且不仅仅出现在其他类别中，同时也会出现在咨询内容页和知识内容页中，但在其他类型(1999001)中占比最高，可达到98.82%。因此需要进一 步分析其类型内部的规律，如以下代码所示。</p>
 1</span> def</span> page199</span>(</span>i</span>)</span>:</span> #</span>自定义统计函数</span></span>
 2</span>     j</span> =</span> i</span>[</span>[</span>'</span>fullURL</span>'</span>,</span>'</span>pageTitle</span>'</span>]</span>]</span>[</span>(</span>i</span>[</span>'</span>fullURLId</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>199</span>'</span>)</span>)</span> &</span> </span></span>
 3</span>          (</span>i</span>[</span>'</span>fullURL</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>\?</span>'</span>)</span>)</span>]</span></span>
 4</span>     j</span>[</span>'</span>pageTitle</span>'</span>]</span>.</span>fillna</span>(</span>'</span>空</span>'</span>,</span>inplace</span>=</span>True</span>)</span></span>
 5</span>     j</span>[</span>'</span>type</span>'</span>]</span> =</span> '</span>其他</span>'</span> #</span> 添加空列</span></span>
 6</span>     j</span>[</span>'</span>type</span>'</span>]</span>[</span>j</span>[</span>'</span>pageTitle</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>法律快车-律师助手</span>'</span>)</span>]</span>=</span> '</span>法律快车-律师助手</span>'</span></span>
 7</span>     j</span>[</span>'</span>type</span>'</span>]</span>[</span>j</span>[</span>'</span>pageTitle</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>咨询发布成功</span>'</span>)</span>]</span>=</span> '</span>咨询发布成功</span>'</span></span>
 8</span>     j</span>[</span>'</span>type</span>'</span>]</span>[</span>j</span>[</span>'</span>pageTitle</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>免费发布法律咨询</span>'</span> )</span>]</span> =</span> '</span>免费发布法律咨询</span>'</span></span>
 9</span>     j</span>[</span>'</span>type</span>'</span>]</span>[</span>j</span>[</span>'</span>pageTitle</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>法律快搜</span>'</span>)</span>]</span> =</span> '</span>快搜</span>'</span></span>
10</span>     j</span>[</span>'</span>type</span>'</span>]</span>[</span>j</span>[</span>'</span>pageTitle</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>法律快车法律经验</span>'</span>)</span>]</span> =</span> '</span>法律快车法律经验</span>'</span></span>
11</span>     j</span>[</span>'</span>type</span>'</span>]</span>[</span>j</span>[</span>'</span>pageTitle</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>法律快车法律咨询</span>'</span>)</span>]</span> =</span> '</span>法律快车法律咨询</span>'</span></span>
12</span>     j</span>[</span>'</span>type</span>'</span>]</span>[</span>(</span>j</span>[</span>'</span>pageTitle</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>_法律快车</span>'</span>)</span>)</span> |</span> </span></span>
13</span>             (</span>j</span>[</span>'</span>pageTitle</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>-法律快车</span>'</span>)</span>)</span>]</span> =</span> '</span>法律快车</span>'</span></span>
14</span>     j</span>[</span>'</span>type</span>'</span>]</span>[</span>j</span>[</span>'</span>pageTitle</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>空</span>'</span>)</span>]</span> =</span> '</span>空</span>'</span></span>
15</span>     </span></span>
16</span>     return</span> j</span></span>
17</span> </span></span>
18</span> #</span> 注意：获取一次sql对象就需要重新访问一下数据库</span></span>
19</span> #</span>engine = create_engine('mysql+pymysql://root:123456@127.0.0.1:3306/test?charset=utf8')</span></span>
20</span> sql</span> =</span> pd</span>.</span>read_sql</span>(</span>'</span>all_gzdata</span>'</span>,</span> engine</span>,</span> chunksize</span> =</span> 10000</span>)</span>#</span> 分块读取数据库信息</span></span>
21</span> #</span>sql = pd.read_sql_query('select * from all_gzdata limit 10000', con=engine)</span></span>
22</span> </span></span>
23</span> counts4</span> =</span> [</span>page199</span>(</span>i</span>)</span> for</span> i</span> in</span> sql</span>]</span> #</span> 逐块统计</span></span>
24</span> counts4</span> =</span> pd</span>.</span>concat</span>(</span>counts4</span>)</span></span>
25</span> d1</span> =</span> counts4</span>[</span>'</span>type</span>'</span>]</span>.</span>value_counts</span>(</span>)</span></span>
26</span> print</span>(</span>d1</span>)</span></span>
27</span> d2</span> =</span> counts4</span>[</span>counts4</span>[</span>'</span>type</span>'</span>]</span>==</span>'</span>其他</span>'</span>]</span></span>
28</span> print</span>(</span>d2</span>)</span></span>
29</span> #</span> 求各个部分的占比并保存数据</span></span>
30</span> df1_</span> =</span>  pd</span>.</span>DataFrame</span>(</span>d1</span>)</span></span>
31</span> df1_</span>[</span>'</span>perc</span>'</span>]</span> =</span> df1_</span>[</span>'</span>type</span>'</span>]</span>/</span>df1_</span>[</span>'</span>type</span>'</span>]</span>.</span>sum</span>(</span>)</span>*</span>100</span></span>
32</span> df1_</span>.</span>sort_values</span>(</span>by</span>=</span>'</span>type</span>'</span>,</span>ascending</span>=</span>False</span>,</span>inplace</span>=</span>True</span>)</span></span>
33</span> print</span>(</span>df1_</span>)</span></span></code></pre>
通过输出结果可以看出，在1999001类型中，标题为“快车—律师助手”这类信息占比为77.09%，这类页面是律师的登录页面。标题为“咨询 发布成功”类信息占比为8.07%，这类页面是自动跳转页面。其他类型的页面大部分为http://www.****.com/ask/question_9152354.html?&from=androidqq类型的网页。根据业务了解该类网页为被分享过的网 页，这类网页需要对其进行处理，处理方式为截取网址中“?”前面的网址并还原网址类型。</p>
在采取截取网址“?”前面网址的处理过程中发现，快搜网址中的类型混杂，不能直接处理。考虑其数据集占比较小，所以在数据处理环节对这部分数据进行删除。同时分析其他类型的网址得出，不包含主网址和关键字的网址有359条记录，如http://www.baidu.com/link？url=O7iBD2KmoJdkHWTZHagDXrxfBFM0AwLmpid12j2d_aejNfq6bwSBe1Ov2jWOFMpIt5XUpXGmNiLDlGg0rMCwstskhB5ftAYtO2_voEnu。</p>
访问记录中有一部分用户并没有点击具体的网页，这类网页以“.html”后缀结尾，且大部分是目录网页，这样的用户可以称为“瞎逛”，漫无目的，总共有165654条记录，统计过程如以下代码所示。</p>
1</span> #</span>统计无目的浏览用户中各个类型占比</span></span>
 2</span> def</span> xiaguang</span>(</span>i</span>)</span>:</span> #</span>自定义统计函数</span></span>
 3</span>     j</span> =</span> i</span>.</span>loc</span>[</span>(</span>i</span>[</span>'</span>fullURL</span>'</span>]</span>.</span>str</span>.</span>contains</span>(</span>'</span>\.html</span>'</span>)</span>)</span>==</span>False</span>,</span></span>
 4</span>               [</span>'</span>fullURL</span>'</span>,</span>'</span>fullURLId</span>'</span>,</span>'</span>pageTitle</span>'</span>]</span>]</span></span>
 5</span>     return</span> j</span></span>
 6</span> </span></span>
 7</span> #</span> 注意获取一次sql对象就需要重新访问一下数据库</span></span>
 8</span> engine</span> =</span> create_engine</span>(</span>'</span>mysql+pymysql://root:123456@127.0.0.1:3306/test1?charset=utf8</span>'</span>)</span></span>
 9</span> sql</span> =</span> pd</span>.</span>read_sql</span>(</span>'</span>all_gzdata</span>'</span>,</span> engine</span>,</span> chunksize</span> =</span> 10000</span>)</span>#</span> 分块读取数据库信息</span></span>
10</span> </span></span>
11</span> counts5</span> =</span> [</span>xiaguang</span>(</span>i</span>)</span> for</span> i</span> in</span> sql</span>]</span></span>
12</span> counts5</span> =</span> pd</span>.</span>concat</span>(</span>counts5</span>)</span></span>
13</span> </span></span>
14</span> xg1</span> =</span> counts5</span>[</span>'</span>fullURLId</span>'</span>]</span>.</span>value_counts</span>(</span>)</span></span>
15</span> print</span>(</span>xg1</span>)</span></span>
16</span> #</span> 求各个部分的占比</span></span>
17</span> xg_</span> =</span>  pd</span>.</span>DataFrame</span>(</span>xg1</span>)</span></span>
18</span> xg_</span>.</span>reset_index</span>(</span>inplace</span>=</span>True</span>)</span></span>
19</span> xg_</span>.</span>columns</span>=</span> [</span>'</span>index</span>'</span>,</span> '</span>num</span>'</span>]</span></span>
20</span> xg_</span>[</span>'</span>perc</span>'</span>]</span> =</span> xg_</span>[</span>'</span>num</span>'</span>]</span>/</span>xg_</span>[</span>'</span>num</span>'</span>]</span>.</span>sum</span>(</span>)</span>*</span>100</span></span>
21</span> xg_</span>.</span>sort_values</span>(</span>by</span>=</span>'</span>num</span>'</span>,</span>ascending</span>=</span>False</span>,</span>inplace</span>=</span>True</span>)</span></span>
22</span> </span></span>
23</span> xg_</span>[</span>'</span>type</span>'</span>]</span> =</span> xg_</span>[</span>'</span>index</span>'</span>]</span>.</span>str</span>.</span>extract</span>(</span>'</span>(\d</span>{3}</span>)</span>'</span>)</span> #</span>提取前三个数字作为类别id    </span></span>
24</span> </span></span>
25</span> xgs_</span> =</span> xg_</span>[</span>[</span>'</span>type</span>'</span>,</span> '</span>num</span>'</span>]</span>]</span>.</span>groupby</span>(</span>'</span>type</span>'</span>)</span>.</span>sum</span>(</span>)</span> #</span>按类别合并</span></span>
26</span> xgs_</span>.</span>sort_values</span>(</span>by</span>=</span>'</span>num</span>'</span>,</span> ascending</span>=</span>False</span>,</span>inplace</span>=</span>True</span>)</span> #</span>降序排列</span></span>
27</span> xgs_</span>[</span>'</span>percentage</span>'</span>]</span> =</span> xgs_</span>[</span>'</span>num</span>'</span>]</span>/</span>xgs_</span>[</span>'</span>num</span>'</span>]</span>.</span>sum</span>(</span>)</span>*</span>100</span></span>
28</span> </span></span>
29</span> print</span>(</span>xgs_</span>.</span>round</span>(</span>4</span>)</span>)</span></span></code></pre>
通过输出结果可以看出，小部分网页类型是与知识、咨询相关的，大部分网页类型是与地区、律师和事物所相关的，这类用户可能是找律师服务的，也可能是“瞎逛”的。</p>
综合以上分析，得到一些与分析目标无关数据的规则，记录这些规则有利于在数据清洗阶段对数据进行清洗操作。</p>
1)咨询发布成功页面。 2)中间类型网页(带有midques_关键字)。 3)网址中带有“?”类型，无法还原其本身类型的快搜网页。 4)重复数据(同一时间同一用户，访问相同网页)。 5)其他类别的数据(主网址不包含关键字)。 6)无点击“.html”行为的用户记录。 7)律师的行为记录(通过快车—律师助手判断)。</p>
2、分析网页点击次数</a></h4>
统计原始数据中用户浏览网页次数的情况，如以下代码所示。</p>
1</span> #</span> 分析网页点击次数</span></span>
 2</span> #</span> 统计点击次数</span></span>
 3</span> engine</span> =</span> create_engine</span>(</span>'</span>mysql+pymysql://root:123456@127.0.0.1:3306/test1?charset=utf8</span>'</span>)</span></span>
 4</span> sql</span> =</span> pd</span>.</span>read_sql</span>(</span>'</span>all_gzdata</span>'</span>,</span> engine</span>,</span> chunksize</span> =</span> 10000</span>)</span>#</span> 分块读取数据库信息</span></span>
 5</span> </span></span>
 6</span> counts1</span> =</span> [</span>i</span>[</span>'</span>realIP</span>'</span>]</span>.</span>value_counts</span>(</span>)</span> for</span> i</span> in</span> sql</span>]</span> #</span> 分块统计各个IP的出现次数</span></span>
 7</span> counts1</span> =</span> pd</span>.</span>concat</span>(</span>counts1</span>)</span>.</span>groupby</span>(</span>level</span>=</span>0</span>)</span>.</span>sum</span>(</span>)</span> #</span> 合并统计结果，level=0表示按照index分组</span></span>
 8</span> print</span>(</span>counts1</span>)</span></span>
 9</span> </span></span>
10</span> counts1_</span> =</span> pd</span>.</span>DataFrame</span>(</span>counts1</span>)</span></span>
11</span> counts1_</span></span>
12</span> counts1</span>[</span>'</span>realIP</span>'</span>]</span> =</span> counts1</span>.</span>index</span>.</span>tolist</span>(</span>)</span></span>
13</span> </span></span>
14</span> counts1_</span>[</span>1</span>]</span>=</span>1</span>  #</span> 添加1列全为1</span></span>
15</span> hit_count</span> =</span> counts1_</span>.</span>groupby</span>(</span>'</span>realIP</span>'</span>)</span>.</span>sum</span>(</span>)</span>  #</span> 统计各个“不同点击次数”分别出现的次数</span></span>
16</span> #</span> 也可以使用counts1_['realIP'].value_counts()功能</span></span>
17</span> hit_count</span>.</span>columns</span>=</span>[</span>'</span>用户数</span>'</span>]</span></span>
18</span> hit_count</span>.</span>index</span>.</span>name</span> =</span> '</span>点击次数</span>'</span></span>
19</span> </span></span>
20</span> #</span> 统计1~7次、7次以上的用户人数</span></span>
21</span> hit_count</span>.</span>sort_index</span>(</span>inplace</span> =</span> True</span>)</span></span>
22</span> hit_count_7</span> =</span> hit_count</span>.</span>iloc</span>[</span>:</span>7</span>,</span>:</span>]</span></span>
23</span> time</span> =</span> hit_count</span>.</span>iloc</span>[</span>7</span>:</span>,</span>0</span>]</span>.</span>sum</span>(</span>)</span>  #</span> 统计点击次数7次以上的用户数</span></span>
24</span> hit_count_7</span> =</span> hit_count_7</span>.</span>append</span>(</span>[</span>{</span>'</span>用户数</span>'</span>:</span>time</span>}</span>]</span>,</span> ignore_index</span>=</span>True</span>)</span></span>
25</span> hit_count_7</span>.</span>index</span> =</span> [</span>'</span>1</span>'</span>,</span>'</span>2</span>'</span>,</span>'</span>3</span>'</span>,</span>'</span>4</span>'</span>,</span>'</span>5</span>'</span>,</span>'</span>6</span>'</span>,</span>'</span>7</span>'</span>,</span>'</span>7次以上</span>'</span>]</span></span>
26</span> hit_count_7</span>[</span>'</span>用户比例</span>'</span>]</span> =</span> hit_count_7</span>[</span>'</span>用户数</span>'</span>]</span> /</span> hit_count_7</span>[</span>'</span>用户数</span>'</span>]</span>.</span>sum</span>(</span>)</span></span>
27</span> print</span>(</span>hit_count_7</span>)</span></span></code></pre>
根据统计结果可知，浏览一次的用户最多，占所有用户的58%左右。</p>
分析浏览次数为一次的用户的行为，如以下代码。</p>
1</span> #</span> 分析浏览一次的用户行为</span></span>
 2</span> </span></span>
 3</span> engine</span> =</span> create_engine</span>(</span>'</span>mysql+pymysql://root:123456@127.0.0.1:3306/test1?charset=utf8</span>'</span>)</span></span>
 4</span> all_gzdata</span> =</span> pd</span>.</span>read_sql_table</span>(</span>'</span>all_gzdata</span>'</span>,</span> con</span> =</span> engine</span>)</span>  #</span> 读取all_gzdata数据</span></span>
 5</span> </span></span>
 6</span> #</span>对realIP进行统计</span></span>
 7</span> #</span> 提取浏览1次网页的数据</span></span>
 8</span> real_count</span> =</span> pd</span>.</span>DataFrame</span>(</span>all_gzdata</span>.</span>groupby</span>(</span>"</span>realIP</span>"</span>)</span>[</span>"</span>realIP</span>"</span>]</span>.</span>count</span>(</span>)</span>)</span></span>
 9</span> real_count</span>.</span>columns</span> =</span> [</span>"</span>count</span>"</span>]</span></span>
10</span> real_count</span>[</span>"</span>realIP</span>"</span>]</span> =</span> real_count</span>.</span>index</span>.</span>tolist</span>(</span>)</span></span>
11</span> user_one</span> =</span> real_count</span>[</span>(</span>real_count</span>[</span>"</span>count</span>"</span>]</span> ==</span> 1</span>)</span>]</span>  #</span> 提取只登录一次的用户</span></span>
12</span> #</span> 通过realIP与原始数据合并</span></span>
13</span> real_one</span> =</span> pd</span>.</span>merge</span>(</span>user_one</span>,</span> all_gzdata</span>,</span> left_on</span>=</span>"</span>realIP</span>"</span>,</span> right_on</span>=</span>"</span>realIP</span>"</span>)</span></span>
14</span> </span></span>
15</span> #</span> 统计浏览一次的网页类型</span></span>
16</span> URL_count</span> =</span> pd</span>.</span>DataFrame</span>(</span>real_one</span>.</span>groupby</span>(</span>"</span>fullURLId</span>"</span>)</span>[</span>"</span>fullURLId</span>"</span>]</span>.</span>count</span>(</span>)</span>)</span></span>
17</span> URL_count</span>.</span>columns</span> =</span> [</span>"</span>count</span>"</span>]</span></span>
18</span> URL_count</span>.</span>sort_values</span>(</span>by</span>=</span>'</span>count</span>'</span>,</span> ascending</span>=</span>False</span>,</span> inplace</span>=</span>True</span>)</span>  #</span> 降序排列</span></span>
19</span> #</span> 统计排名前4和其他的网页类型</span></span>
20</span> URL_count_4</span> =</span> URL_count</span>.</span>iloc</span>[</span>:</span>4</span>,</span>:</span>]</span></span>
21</span> time</span> =</span> hit_count</span>.</span>iloc</span>[</span>4</span>:</span>,</span>0</span>]</span>.</span>sum</span>(</span>)</span>  #</span> 统计其他的</span></span>
22</span> URLindex</span> =</span> URL_count_4</span>.</span>index</span>.</span>values</span></span>
23</span> URL_count_4</span> =</span> URL_count_4</span>.</span>append</span>(</span>[</span>{</span>'</span>count</span>'</span>:</span>time</span>}</span>]</span>,</span> ignore_index</span>=</span>True</span>)</span></span>
24</span> URL_count_4</span>.</span>index</span> =</span> [</span>URLindex</span>[</span>0</span>]</span>,</span> URLindex</span>[</span>1</span>]</span>,</span> URLindex</span>[</span>2</span>]</span>,</span> URLindex</span>[</span>3</span>]</span>,</span> </span></span>
25</span>                      '</span>其他</span>'</span>]</span></span>
26</span> URL_count_4</span>[</span>'</span>比例</span>'</span>]</span> =</span> URL_count_4</span>[</span>'</span>count</span>'</span>]</span> /</span> URL_count_4</span>[</span>'</span>count</span>'</span>]</span>.</span>sum</span>(</span>)</span></span>
27</span> print</span>(</span>URL_count_4</span>)</span></span></code></pre>（四）数据预处理</a></h3>
1、删除不符合规则的网页</a></h4>
通过分析原始数据发现，不符合规则的网页包括中间页面的网址、咨询发布成功页面、律师登录助手页面等，需要对其进行删除处理，如以下代码所示。</p>
</span>
1</span> #</span> 删除不符合规则的网页</span></span>
 2</span> import</span> os</span></span>
 3</span> import</span> re</span></span>
 4</span> import</span> pandas</span> as</span> pd</span></span>
 5</span> import</span> pymysql</span> as</span> pm</span></span>
 6</span> from</span> random</span> import</span> sample</span></span>
 7</span> </span></span>
 8</span> #</span> 修改工作路径到指定文件夹</span></span>
 9</span> os</span>.</span>chdir</span>(</span>"</span>E:/大三下/数据分析/数据/第十一章</span>"</span>)</span></span>
10</span> </span></span>
11</span> #</span> 读取数据</span></span>
12</span> con</span> =</span> pm</span>.</span>connect</span>(</span>host</span> =</span> '</span>localhost</span>'</span>,</span>user</span> =</span> '</span>root</span>'</span>,</span>password</span> =</span> '</span>123456</span>'</span>,</span>database</span> =</span> '</span>test1</span>'</span>,</span>charset</span>=</span>'</span>utf8</span>'</span>)</span></span>
13</span> data</span> =</span> pd</span>.</span>read_sql</span>(</span>'</span>select * from all_gzdata</span>'</span>,</span>con</span>=</span>con</span>)</span></span>
14</span> con</span>.</span>close</span>(</span>)</span>  #</span> 关闭连接</span></span>
15</span> </span></span>
16</span> #</span> 取出107类型数据</span></span>
17</span> index107</span> =</span> [</span>re</span>.</span>search</span>(</span>'</span>107</span>'</span>,</span>str</span>(</span>i</span>)</span>)</span>!=</span>None</span> for</span> i</span> in</span> data</span>.</span>loc</span>[</span>:</span>,</span>'</span>fullURLId</span>'</span>]</span>]</span></span>
18</span> data_107</span> =</span> data</span>.</span>loc</span>[</span>index107</span>,</span>:</span>]</span></span>
19</span> </span></span>
20</span> #</span> 在107类型中筛选出婚姻类数据</span></span>
21</span> index</span> =</span> [</span>re</span>.</span>search</span>(</span>'</span>hunyin</span>'</span>,</span>str</span>(</span>i</span>)</span>)</span>!=</span>None</span> for</span> i</span> in</span> data_107</span>.</span>loc</span>[</span>:</span>,</span>'</span>fullURL</span>'</span>]</span>]</span></span>
22</span> data_hunyin</span> =</span> data_107</span>.</span>loc</span>[</span>index</span>,</span>:</span>]</span></span>
23</span> </span></span>
24</span> #</span> 提取所需字段(realIP、fullURL)</span></span>
25</span> info</span> =</span> data_hunyin</span>.</span>loc</span>[</span>:</span>,</span>[</span>'</span>realIP</span>'</span>,</span>'</span>fullURL</span>'</span>]</span>]</span></span>
26</span> </span></span>
27</span> #</span> 去除网址中“？”及其后面内容</span></span>
28</span> da</span> =</span> [</span>re</span>.</span>sub</span>(</span>'</span>\?.*</span>'</span>,</span>'</span>'</span>,</span>str</span>(</span>i</span>)</span>)</span> for</span> i</span> in</span> info</span>.</span>loc</span>[</span>:</span>,</span>'</span>fullURL</span>'</span>]</span>]</span></span>
29</span> info</span>.</span>loc</span>[</span>:</span>,</span>'</span>fullURL</span>'</span>]</span> =</span> da</span>     #</span> 将info中‘fullURL’那列换成da</span></span>
30</span> #</span> 去除无html网址</span></span>
31</span> index</span> =</span> [</span>re</span>.</span>search</span>(</span>'</span>\.html</span>'</span>,</span>str</span>(</span>i</span>)</span>)</span>!=</span>None</span> for</span> i</span> in</span> info</span>.</span>loc</span>[</span>:</span>,</span>'</span>fullURL</span>'</span>]</span>]</span></span>
32</span> index</span>.</span>count</span>(</span>True</span>)</span>   #</span> True 或者 1 ， False 或者 0</span></span>
33</span> info1</span> =</span> info</span>.</span>loc</span>[</span>index</span>,</span>:</span>]</span></span></code></pre>
清洗后数据依旧存在大量的目录网页（可理解为用户浏览信息的路径），这类网页不但对构建推荐系统没有作用，反而会影响推荐结果的准确性，同样需要处理。</p>
2、还原翻页网址</a></h4>
本案例主要对知识相关的网页类型数据进行分析。处理翻页情况最直接的方法是将翻页的网址删掉，但是用户是通过搜索引擎进入网站的，访问入口不一定是原始页面，采取删除方法会损失大量有效数据，影响推荐结果。因此对该类网页的处理方式是:首先识别翻页的网址，然后对翻页的网址进行还原，最后针对每个用户访问的页面进行去重操作，如以下代码所示。</p>
1</span> #</span> 找出翻页和非翻页网址</span></span>
 2</span> index</span> =</span> [</span>re</span>.</span>search</span>(</span>'</span>/\d+_\d+\.html</span>'</span>,</span>i</span>)</span>!=</span>None</span> for</span> i</span> in</span> info1</span>.</span>loc</span>[</span>:</span>,</span>'</span>fullURL</span>'</span>]</span>]</span></span>
 3</span> index1</span> =</span> [</span>i</span>==</span>False</span> for</span> i</span> in</span> index</span>]</span></span>
 4</span> info1_1</span> =</span> info1</span>.</span>loc</span>[</span>index</span>,</span>:</span>]</span>   #</span> 带翻页网址</span></span>
 5</span> info1_2</span> =</span> info1</span>.</span>loc</span>[</span>index1</span>,</span>:</span>]</span>  #</span> 无翻页网址</span></span>
 6</span> #</span> 将翻页网址还原</span></span>
 7</span> da</span> =</span> [</span>re</span>.</span>sub</span>(</span>'</span>_\d+\.html</span>'</span>,</span>'</span>.html</span>'</span>,</span>str</span>(</span>i</span>)</span>)</span> for</span> i</span> in</span> info1_1</span>.</span>loc</span>[</span>:</span>,</span>'</span>fullURL</span>'</span>]</span>]</span></span>
 8</span> info1_1</span>.</span>loc</span>[</span>:</span>,</span>'</span>fullURL</span>'</span>]</span> =</span> da</span></span>
 9</span> #</span> 翻页与非翻页网址合并</span></span>
10</span> frames</span> =</span> [</span>info1_1</span>,</span>info1_2</span>]</span></span>
11</span> info2</span> =</span> pd</span>.</span>concat</span>(</span>frames</span>)</span></span>
12</span> #</span> 或者</span></span>
13</span> info2</span> =</span> pd</span>.</span>concat</span>(</span>[</span>info1_1</span>,</span>info1_2</span>]</span>,</span>axis</span> =</span> 0</span>)</span>   #</span> 默认为0，即行合并</span></span>
14</span> #</span> 去重（realIP和fullURL两列相同）</span></span>
15</span> info3</span> =</span> info2</span>.</span>drop_duplicates</span>(</span>)</span></span>
16</span> #</span> 将IP转换成字符型数据</span></span>
17</span> info3</span>.</span>iloc</span>[</span>:</span>,</span>0</span>]</span> =</span> [</span>str</span>(</span>index</span>)</span> for</span> index</span> in</span> info3</span>.</span>iloc</span>[</span>:</span>,</span>0</span>]</span>]</span></span>
18</span> info3</span>.</span>iloc</span>[</span>:</span>,</span>1</span>]</span> =</span> [</span>str</span>(</span>index</span>)</span> for</span> index</span> in</span> info3</span>.</span>iloc</span>[</span>:</span>,</span>1</span>]</span>]</span></span>
19</span> len</span>(</span>info3</span>)</span></span></code></pre>3、筛去浏览次数不满两次的用户</a></h4>
根据数据探索的结果可知，数据中存在大量仅浏览一次就跳出的用户，浏览次数在两次及以上的用户的浏览记录更适于推荐，而浏览次数仅一次的用户的浏览记录进入推荐模型会影响推荐模型的效果，因此需要筛去浏览次数不满两次的用户，如以下代码所示。</p>
1</span> #</span> 筛选满足一定浏览次数的IP</span></span>
 2</span> IP_count</span> =</span> info3</span>[</span>'</span>realIP</span>'</span>]</span>.</span>value_counts</span>(</span>)</span></span>
 3</span> #</span> 找出IP集合</span></span>
 4</span> IP</span> =</span> list</span>(</span>IP_count</span>.</span>index</span>)</span></span>
 5</span> count</span> =</span> list</span>(</span>IP_count</span>.</span>values</span>)</span></span>
 6</span> #</span> 统计每个IP的浏览次数，并存放进IP_count数据框中,第一列为IP，第二列为浏览次数</span></span>
 7</span> IP_count</span> =</span> pd</span>.</span>DataFrame</span>(</span>{</span>'</span>IP</span>'</span>:</span>IP</span>,</span>'</span>count</span>'</span>:</span>count</span>}</span>)</span></span>
 8</span> #</span> 3.3筛选出浏览网址在n次以上的IP集合</span></span>
 9</span> n</span> =</span> 2</span></span>
10</span> index</span> =</span> IP_count</span>.</span>loc</span>[</span>:</span>,</span>'</span>count</span>'</span>]</span>></span>n</span></span>
11</span> IP_index</span> =</span> IP_count</span>.</span>loc</span>[</span>index</span>,</span>'</span>IP</span>'</span>]</span></span></code></pre>4、划分数据集</a></h4>
将数据集按8：2的比例划分为训练集和测试集。</p>
1</span> #</span> 划分IP集合为训练集和测试集</span></span>
 2</span> index_tr</span> =</span> sample</span>(</span>range</span>(</span>0</span>,</span>len</span>(</span>IP_index</span>)</span>)</span>,</span>int</span>(</span>len</span>(</span>IP_index</span>)</span>*</span>0.8</span>)</span>)</span>  #</span> 或者np.random.sample</span></span>
 3</span> index_te</span> =</span> [</span>i</span> for</span> i</span> in</span> range</span>(</span>0</span>,</span>len</span>(</span>IP_index</span>)</span>)</span> if</span> i</span> not</span> in</span> index_tr</span>]</span></span>
 4</span> IP_tr</span> =</span> IP_index</span>[</span>index_tr</span>]</span></span>
 5</span> IP_te</span> =</span> IP_index</span>[</span>index_te</span>]</span></span>
 6</span> #</span> 将对应数据集划分为训练集和测试集</span></span>
 7</span> index_tr</span> =</span> [</span>i</span> in</span> list</span>(</span>IP_tr</span>)</span> for</span> i</span> in</span> info3</span>.</span>loc</span>[</span>:</span>,</span>'</span>realIP</span>'</span>]</span>]</span></span>
 8</span> index_te</span> =</span> [</span>i</span> in</span> list</span>(</span>IP_te</span>)</span> for</span> i</span> in</span> info3</span>.</span>loc</span>[</span>:</span>,</span>'</span>realIP</span>'</span>]</span>]</span></span>
 9</span> data_tr</span> =</span> info3</span>.</span>loc</span>[</span>index_tr</span>,</span>:</span>]</span></span>
10</span> data_te</span> =</span> info3</span>.</span>loc</span>[</span>index_te</span>,</span>:</span>]</span></span>
11</span> print</span>(</span>len</span>(</span>data_tr</span>)</span>)</span></span>
12</span> IP_tr</span> =</span> data_tr</span>.</span>iloc</span>[</span>:</span>,</span>0</span>]</span>  #</span> 训练集IP</span></span>
13</span> url_tr</span> =</span> data_tr</span>.</span>iloc</span>[</span>:</span>,</span>1</span>]</span>  #</span> 训练集网址</span></span>
14</span> IP_tr</span> =</span> list</span>(</span>set</span>(</span>IP_tr</span>)</span>)</span>  #</span> 去重处理</span></span>
15</span> url_tr</span> =</span> list</span>(</span>set</span>(</span>url_tr</span>)</span>)</span>  #</span> 去重处理</span></span>
16</span> len</span>(</span>url_tr</span>)</span></span></code></pre>（五）构建智能推荐模型</a></h3>
1、模型构建</a></h4>
将训练集中的数据转换成0-1二元型数据，使用ItemCF算法对数据进行建模，并给出预测推荐结果。</p>
1</span> import</span> pandas</span> as</span> pd</span></span>
 2</span> #</span> 利用训练集数据构建模型</span></span>
 3</span> UI_matrix_tr</span> =</span> pd</span>.</span>DataFrame</span>(</span>0</span>,</span>index</span>=</span>IP_tr</span>,</span>columns</span>=</span>url_tr</span>)</span></span>
 4</span> #</span> 求用户－物品矩阵</span></span>
 5</span> for</span> i</span> in</span> data_tr</span>.</span>index</span>:</span></span>
 6</span>     UI_matrix_tr</span>.</span>loc</span>[</span>data_tr</span>.</span>loc</span>[</span>i</span>,</span>'</span>realIP</span>'</span>]</span>,</span>data_tr</span>.</span>loc</span>[</span>i</span>,</span>'</span>fullURL</span>'</span>]</span>]</span> =</span> 1</span></span>
 7</span> sum</span>(</span>UI_matrix_tr</span>.</span>sum</span>(</span>axis</span>=</span>1</span>)</span>)</span></span>
 8</span> </span></span>
 9</span> #</span> 求物品相似度矩阵（因计算量较大，需要耗费的时间较久）</span></span>
10</span> Item_matrix_tr</span> =</span> pd</span>.</span>DataFrame</span>(</span>0</span>,</span>index</span>=</span>url_tr</span>,</span>columns</span>=</span>url_tr</span>)</span></span>
11</span> for</span> i</span> in</span> Item_matrix_tr</span>.</span>index</span>:</span></span>
12</span>     for</span> j</span> in</span> Item_matrix_tr</span>.</span>index</span>:</span></span>
13</span>         a</span> =</span> sum</span>(</span>UI_matrix_tr</span>.</span>loc</span>[</span>:</span>,</span>[</span>i</span>,</span>j</span>]</span>]</span>.</span>sum</span>(</span>axis</span>=</span>1</span>)</span>==</span>2</span>)</span></span>
14</span>         b</span> =</span> sum</span>(</span>UI_matrix_tr</span>.</span>loc</span>[</span>:</span>,</span>[</span>i</span>,</span>j</span>]</span>]</span>.</span>sum</span>(</span>axis</span>=</span>1</span>)</span>!=</span>0</span>)</span></span>
15</span>         Item_matrix_tr</span>.</span>loc</span>[</span>i</span>,</span>j</span>]</span> =</span> a</span>/</span>b</span></span>
16</span> </span></span>
17</span> #</span> 将物品相似度矩阵对角线处理为零</span></span>
18</span> for</span> i</span> in</span> Item_matrix_tr</span>.</span>index</span>:</span></span>
19</span>     Item_matrix_tr</span>.</span>loc</span>[</span>i</span>,</span>i</span>]</span>=</span>0</span></span>
20</span> </span></span>
21</span> #</span> 利用测试集数据对模型评价</span></span>
22</span> IP_te</span> =</span> data_te</span>.</span>iloc</span>[</span>:</span>,</span>0</span>]</span></span>
23</span> url_te</span> =</span> data_te</span>.</span>iloc</span>[</span>:</span>,</span>1</span>]</span></span>
24</span> IP_te</span> =</span> list</span>(</span>set</span>(</span>IP_te</span>)</span>)</span></span>
25</span> url_te</span> =</span> list</span>(</span>set</span>(</span>url_te</span>)</span>)</span></span>
26</span> </span></span>
27</span> #</span> 测试集数据用户物品矩阵</span></span>
28</span> UI_matrix_te</span> =</span> pd</span>.</span>DataFrame</span>(</span>0</span>,</span>index</span>=</span>IP_te</span>,</span>columns</span>=</span>url_te</span>)</span></span>
29</span> for</span> i</span> in</span> data_te</span>.</span>index</span>:</span></span>
30</span>     UI_matrix_te</span>.</span>loc</span>[</span>data_te</span>.</span>loc</span>[</span>i</span>,</span>'</span>realIP</span>'</span>]</span>,</span>data_te</span>.</span>loc</span>[</span>i</span>,</span>'</span>fullURL</span>'</span>]</span>]</span> =</span> 1</span></span>
31</span> </span></span>
32</span> #</span> 对测试集IP进行推荐</span></span>
33</span> Res</span> =</span> pd</span>.</span>DataFrame</span>(</span>'</span>NaN</span>'</span>,</span>index</span>=</span>data_te</span>.</span>index</span>,</span></span>
34</span>                    columns</span>=</span>[</span>'</span>IP</span>'</span>,</span>'</span>已浏览网址</span>'</span>,</span>'</span>推荐网址</span>'</span>,</span>'</span>T/F</span>'</span>]</span>)</span></span>
35</span> Res</span>.</span>loc</span>[</span>:</span>,</span>'</span>IP</span>'</span>]</span>=</span>list</span>(</span>data_te</span>.</span>iloc</span>[</span>:</span>,</span>0</span>]</span>)</span></span>
36</span> Res</span>.</span>loc</span>[</span>:</span>,</span>'</span>已浏览网址</span>'</span>]</span>=</span>list</span>(</span>data_te</span>.</span>iloc</span>[</span>:</span>,</span>1</span>]</span>)</span></span>
37</span> </span></span>
38</span> #</span> 开始推荐</span></span>
39</span> for</span> i</span> in</span> Res</span>.</span>index</span>:</span></span>
40</span>     if</span> Res</span>.</span>loc</span>[</span>i</span>,</span>'</span>已浏览网址</span>'</span>]</span> in</span> list</span>(</span>Item_matrix_tr</span>.</span>index</span>)</span>:</span></span>
41</span>         Res</span>.</span>loc</span>[</span>i</span>,</span>'</span>推荐网址</span>'</span>]</span> =</span> Item_matrix_tr</span>.</span>loc</span>[</span>Res</span>.</span>loc</span>[</span>i</span>,</span>'</span>已浏览网址</span>'</span>]</span>,</span></span>
42</span>                 :</span>]</span>.</span>argmax</span>(</span>)</span></span>
43</span>         if</span> Res</span>.</span>loc</span>[</span>i</span>,</span>'</span>推荐网址</span>'</span>]</span> in</span> url_te</span>:</span></span>
44</span>             Res</span>.</span>loc</span>[</span>i</span>,</span>'</span>T/F</span>'</span>]</span>=</span>UI_matrix_te</span>.</span>loc</span>[</span>Res</span>.</span>loc</span>[</span>i</span>,</span>'</span>IP</span>'</span>]</span>,</span></span>
45</span>                     Res</span>.</span>loc</span>[</span>i</span>,</span>'</span>推荐网址</span>'</span>]</span>]</span>==</span>1</span></span>
46</span>         else</span>:</span></span>
47</span>             Res</span>.</span>loc</span>[</span>i</span>,</span>'</span>T/F</span>'</span>]</span> =</span> False</span></span>
48</span> </span></span>
49</span> #</span> 保存推荐结果</span></span>
50</span> Res</span>.</span>to_csv</span>(</span>'</span>E:/大三下/数据分析/数据/第十一章/Res.csv</span>'</span>,</span>index</span>=</span>False</span>,</span>encoding</span>=</span>'</span>utf8</span>'</span>)</span></span></code></pre>2、模型评价</a></h4>
计算推荐结果的准确率、召回率和F1指标。</p>
 1</span> import</span> pandas</span> as</span> pd</span></span>
 2</span> #</span> 读取保存的推荐结果</span></span>
 3</span> Res</span> =</span> pd</span>.</span>read_csv</span>(</span>'</span>./tmp/Res.csv</span>'</span>,</span>keep_default_na</span>=</span>False</span>,</span> encoding</span>=</span>'</span>utf8</span>'</span>)</span></span>
 4</span> </span></span>
 5</span> #</span> 计算推荐准确率</span></span>
 6</span> Pre</span> =</span> round</span>(</span>sum</span>(</span>Res</span>.</span>loc</span>[</span>:</span>,</span>'</span>T/F</span>'</span>]</span>==</span>'</span>True</span>'</span>)</span> /</span> (</span>len</span>(</span>Res</span>.</span>index</span>)</span>-</span>sum</span>(</span>Res</span>.</span>loc</span>[</span>:</span>,</span>'</span>T/F</span>'</span>]</span>==</span>'</span>NaN</span>'</span>)</span>)</span>,</span> 3</span>)</span></span>
 7</span> </span></span>
 8</span> print</span>(</span>Pre</span>)</span></span>
 9</span> </span></span>
10</span> #</span> 计算推荐召回率</span></span>
11</span> Rec</span> =</span> round</span>(</span>sum</span>(</span>Res</span>.</span>loc</span>[</span>:</span>,</span>'</span>T/F</span>'</span>]</span>==</span>'</span>True</span>'</span>)</span> /</span> (</span>sum</span>(</span>Res</span>.</span>loc</span>[</span>:</span>,</span>'</span>T/F</span>'</span>]</span>==</span>'</span>True</span>'</span>)</span>+</span>sum</span>(</span>Res</span>.</span>loc</span>[</span>:</span>,</span>'</span>T/F</span>'</span>]</span>==</span>'</span>NaN</span>'</span>)</span>)</span>,</span> 3</span>)</span></span>
12</span> </span></span>
13</span> print</span>(</span>Rec</span>)</span></span>
14</span> </span></span>
15</span> #</span> 计算F1指标</span></span>
16</span> F1</span> =</span> round</span>(</span>2</span>*</span>Pre</span>*</span>Rec</span>/</span>(</span>Pre</span>+</span>Rec</span>)</span>,</span>3</span>)</span></span>
17</span> print</span>(</span>F1</span>)</span></span></code></pre>


数据挖掘作业5
Unknown — Sat, 22 Apr 2023 10:38:00 +0800
家用热水器用户行为分析与事件识别</a></h1>
一、背景与挖掘目标</a></h2>
居民在使用家用热水器的过程中，会因为地区气候、不同区域和用户年龄性别差异等原因形成不同的使用习惯。家电企业若能深入了解其产品在不同用户群中的使用习惯，开发符合客户需求和使用习惯的功能，就能开拓新市场。</p>
本案例将依据BP神经网络算法构建洗浴事件识别模型，进而对不同地区的用户的洗浴事件进行识别，然后根据识别结果比较不同客户群的客户使用习惯，以加深对客户需求的理解等。从而厂商便可以对不同的客户群提供最适合的个性化产品，改进新产品的智能化研发并制定相应的营销策略。</p>
国内某热水器生产厂商新研发的一种高端智能热水器，在状态发生改变或者有水流状态时，会采集各监控指标数据。本案例基于热水器采集的时间序列数据，根据水流量和停顿时间间隔，将顺序排列的离散的用水时间节点划分为不同大小的时间区间，每个区间都是一个可理解的一次完整的用水事件，并以热水器一次完整用水事件作为一个基本事件，将时间序列数据划分为独立的用水事件，并识别出其中属于洗浴的事件。基于以上工作，该厂商可从热水器智能操作和节能运行等方面对产品进行优化。</p>
在热水器用户行为分析过程中，用水事件识别是最为关键的环节。根据该热水器生产厂商提供的数据，热水器用户用水事件划分与识别案例的整体目标如下:</p>
1)根据热水器采集到的数据，划分一次完整用水事件。 2)在划分好的一次完整用水事件中，识别出洗浴事件。</p>
二、分析方法与过程</a></h2>
热水器用户用水事件划分与识别案例主要包括以下5个步骤:</p>
1)对热水器用户的历史用水数据进行选择性抽取，构建专家样本。</p>
2)对步骤1形成的数据集，进行数据探索分析与预处理，包括探索水流量的分布情况，删除冗余属性，识别用水数据的缺失值，并对缺失值进行处理，然后根据建模的需要进行属性构造等。最后根据以上处理，对热水器用户用水样本数据建立用水事件时间间隔识别模型和划分 一次完整的用水事件模型，接着在一次完整用水事件划分结果的基础上，剔除短暂用水事件、缩小识别范围等。</p>
3)在步骤2得到的建模样本数据基础上，建立洗浴事件识别模型，对洗浴事件识别模型进行模型分析评价。</p>
4)应用步骤3形成的模型结果，并对洗浴事件划分进行优化。</p>
5)调用洗浴事件识别模型，对实时监控的热水器流水数据进行洗浴事件自动识别。</p>
（一）数据探索分析</a></h3>
本案例对原始数据采用无放回随机抽样法，抽取200家热水器用户自2014年1月1日至2014年12月31日的用水记录作为原始建模数据。由于热水器用户不仅使用热水器来洗浴，还有洗手、洗脸、刷牙、洗菜、做饭等用水行为，所以热水器采集到的数据来自各种不同的用水事件。</p>
热水器采集的用水数据包含12个属性:热水器编码、发生时间、开关机状态、加热中、保温中、有无水流、实际温度、热水量、水流量、节能模式、加热剩余时间和当前设置温度等。</p>
探索分析热水器的水流量状况，其中“有无水流”和“水流量”属性最能直观体现热水器的水流量情况，对这两个属性进行探索分析。</p>
 1</span> import</span> pandas</span> as</span> pd</span></span>
 2</span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span>
 3</span> </span></span>
 4</span> inputfile</span> =</span> '</span>E:/大三下/数据分析/数据/第十章/original_data.xls</span>'</span>  #</span> 输入的数据文件</span></span>
 5</span> data</span> =</span> pd</span>.</span>read_excel</span>(</span>inputfile</span>)</span>  #</span> 读取数据</span></span>
 6</span> </span></span>
 7</span> #</span> 查看有无水流的分布</span></span>
 8</span> #</span> 数据提取</span></span>
 9</span> lv_non</span> =</span> pd</span>.</span>value_counts</span>(</span>data</span>[</span>'</span>有无水流</span>'</span>]</span>)</span>[</span>'</span>无</span>'</span>]</span></span>
10</span> lv_move</span> =</span> pd</span>.</span>value_counts</span>(</span>data</span>[</span>'</span>有无水流</span>'</span>]</span>)</span>[</span>'</span>有</span>'</span>]</span></span>
11</span> #</span> 绘制条形图</span></span>
12</span>  </span></span>
13</span> fig</span> =</span> plt</span>.</span>figure</span>(</span>figsize</span> =</span> (</span>6</span> ,</span>5</span>)</span>)</span>  #</span> 设置画布大小</span></span>
14</span> plt</span>.</span>rcParams</span>[</span>'</span>font.sans-serif</span>'</span>]</span> =</span> '</span>SimHei</span>'</span>  #</span> 设置中文显示</span></span>
15</span> plt</span>.</span>rcParams</span>[</span>'</span>axes.unicode_minus</span>'</span>]</span> =</span> False</span></span>
16</span> plt</span>.</span>bar</span>(</span>x</span>=</span>range</span>(</span>2</span>)</span>,</span> height</span>=</span>[</span>lv_non</span>,</span>lv_move</span>]</span>,</span> width</span>=</span>0.4</span>,</span> alpha</span>=</span>0.8</span>,</span> </span></span>
17</span>         color</span>=</span>'</span>skyblue</span>'</span>)</span></span>
18</span> plt</span>.</span>xticks</span>(</span>[</span>index</span> for</span> index</span> in</span> range</span>(</span>2</span>)</span>]</span>,</span> [</span>'</span>无</span>'</span>,</span>'</span>有</span>'</span>]</span>)</span></span>
19</span> plt</span>.</span>xlabel</span>(</span>'</span>水流状态</span>'</span>)</span></span>
20</span> plt</span>.</span>ylabel</span>(</span>'</span>记录数</span>'</span>)</span></span>
21</span> plt</span>.</span>title</span>(</span>'</span>2019320143322 不同水流状态记录数</span>'</span>)</span></span>
22</span> plt</span>.</span>show</span>(</span>)</span></span>
23</span> plt</span>.</span>close</span>(</span>)</span></span>
24</span> </span></span>
25</span> #</span> 查看水流量分布</span></span>
26</span> water</span> =</span> data</span>[</span>'</span>水流量</span>'</span>]</span></span>
27</span> #</span> 绘制水流量分布箱型图</span></span>
28</span> fig</span> =</span> plt</span>.</span>figure</span>(</span>figsize</span> =</span> (</span>5</span> ,</span>8</span>)</span>)</span></span>
29</span> plt</span>.</span>boxplot</span>(</span>water</span>,</span> </span></span>
30</span>             patch_artist</span>=</span>True</span>,</span></span>
31</span>             labels</span> =</span> [</span>'</span>水流量</span>'</span>]</span>,</span>  #</span> 设置x轴标题</span></span>
32</span>             boxprops</span> =</span> {</span>'</span>facecolor</span>'</span>:</span>'</span>lightblue</span>'</span>}</span>)</span>  #</span> 设置填充颜色</span></span>
33</span> plt</span>.</span>title</span>(</span>'</span>2019320143322 水流量分布箱线图</span>'</span>)</span></span>
34</span> #</span> 显示y坐标轴的底线</span></span>
35</span> plt</span>.</span>grid</span>(</span>axis</span>=</span>'</span>y</span>'</span>)</span></span>
36</span> plt</span>.</span>show</span>(</span>)</span></span></code></pre>
通过不同水流状态的记录条形图可知无水流状态的记录明显比有水流状态的记录要多。通过水流量分布箱型图可知箱体贴近0，说明无水流量的记录较多，水流量的分布与水流状态的分布一致。</p>
“用水停顿时间间隔”定义为一条水流量不为0的流水记录同下一条水流量不为0的流水记录之间的时间间隔。根据现场实验统计，两次用水过程的用水停顿间隔时长一般不大于4分钟。为了探究热水器用户真实用水停顿时间间隔的分布情况，统计用水停顿的时间间隔并做出频率分布表。通过频率分布表分析用户用水停顿时间间隔的规律性，具体的数据如下表所示。</p>
通过分析表10-2可知，停顿时间间隔为0~0.3分钟的频率很高，根据日常用水经验可以判断其为一次用水时间中的停顿;停顿时间间隔为 6~13分钟的频率较低，分析其为两次用水事件之间的停顿。根据现场实验统计用水停顿的时间间隔可知，两次用水事件的停顿时间间隔分布在3~7分钟。</p>
（二）数据预处理</a></h3>
1、属性规约</a></h4>
由于热水器采集的用水数据属性较多，本案例做以下处理。</p>
因为分析的主要对象为热水器用户，分析的主要目标为热水器用户洗浴行为的一般规律，所以“热水器编号”属性可以去除;因为在热水器采集的数据中，“有无水流”属性可以通过“水流量”属性反映出来，“节能模式”属性取值相同均为“关”，对分析无作用，所以可以去除。</p>
以下代码为删除冗余属性“热水器编号”、“有无水流”、“节能模式”。</p>
1</span> import</span> pandas</span> as</span> pd</span></span>
2</span> import</span> numpy</span> as</span> np</span></span>
3</span> data</span> =</span> pd</span>.</span>read_excel</span>(</span>'</span>E:/大三下/数据分析/数据/第十章/original_data.xls</span>'</span>)</span></span>
4</span> print</span>(</span>'</span>初始状态的数据形状为：</span>'</span>,</span> data</span>.</span>shape</span>)</span></span>
5</span> #</span> 删除热水器编号、有无水流、节能模式属性</span></span>
6</span> data</span>.</span>drop</span>(</span>labels</span>=</span>[</span>"</span>热水器编号</span>"</span>,</span>"</span>有无水流</span>"</span>,</span>"</span>节能模式</span>"</span>]</span>,</span>axis</span>=</span>1</span>,</span>inplace</span>=</span>True</span>)</span> </span></span>
7</span> print</span>(</span>'</span>删除冗余属性后的数据形状为：</span>'</span>,</span> data</span>.</span>shape</span>)</span></span>
8</span> data</span>.</span>to_csv</span>(</span>'</span>E:/大三下/数据分析/数据/第十章/water_heart.csv</span>'</span>,</span>index</span>=</span>False</span>)</span></span>
</span></code></pre>2、划分用水事件</a></h4>
热水器用户的用水数据存储在数据库中，记录了各种各样的用水事件，包括洗浴、洗手、刷牙、洗脸、洗衣、洗菜等，而且一次用水事件由数条甚至数千条的状态记录组成。所以本案例首先需要在大量的状态记录中划分出哪些连续的数据是一次完整的用水事件。在用水状态记录中，水流量不为0，表明热水器用户正在使用热水;而水流量为0时，则表明热水器用户用热水时发生停顿或者用热水结束。对于任何一个用水记录，如果它的向前时差超过阈值T，则将它记为事件的开始编号;如果它的向后时差超过阈值T，则将其记为事件的结束编号。</p>
一次完整用水事件的划分步骤如下: 1)读取数据记录，识别所有水流量不为0的状态记录，将它们的发生时间记为序列t1。2)对序列t1构建其向前时差列和向后时差列，并分别与阈值进行比较。向前时差超过阈值T，则将它记为新的用水事件的开始编号;如果向后时差超过阈值T，则将其记为用水事件的结束编号。</p>
循环执行步骤2，直到向前时差列和向后时差列与均值比较完毕， 则结束事件划分。</p>
用水事件划分主要分为两个步骤，即确定单次用水时长间隔，计算两条相邻记录的时间，实现代码如下。</p>
 1</span> import</span> pandas</span> as</span> pd</span></span>
 2</span> import</span> numpy</span> as</span> np</span></span>
 3</span> #</span> 读取数据</span></span>
 4</span> data</span> =</span> pd</span>.</span>read_csv</span>(</span>'</span>E:/大三下/数据分析/数据/第十章/water_heart.csv</span>'</span>)</span></span>
 5</span> #</span> 划分用水事件</span></span>
 6</span> threshold</span> =</span> pd</span>.</span>Timedelta</span>(</span>'</span>4 min</span>'</span>)</span>  #</span> 阈值为4分钟</span></span>
 7</span> data</span>[</span>'</span>发生时间</span>'</span>]</span> =</span> pd</span>.</span>to_datetime</span>(</span>data</span>[</span>'</span>发生时间</span>'</span>]</span>,</span> format</span> =</span> '</span>%Y%m</span>%d</span>%H%M%S</span>'</span>)</span>  #</span> 转换时间格式</span></span>
 8</span> data</span> =</span> data</span>[</span>data</span>[</span>'</span>水流量</span>'</span>]</span> ></span> 0</span>]</span>  #</span> 只要流量大于0的记录</span></span>
 9</span> sjKs</span> =</span> data</span>[</span>'</span>发生时间</span>'</span>]</span>.</span>diff</span>(</span>)</span> ></span> threshold</span>  #</span> 相邻时间向前差分，比较是否大于阈值</span></span>
10</span> sjKs</span>.</span>iloc</span>[</span>0</span>]</span> =</span> True</span>  #</span> 令第一个时间为第一个用水事件的开始事件</span></span>
11</span> sjJs</span> =</span> sjKs</span>.</span>iloc</span>[</span>1</span>:</span>]</span>  #</span> 向后差分的结果</span></span>
12</span> sjJs</span> =</span> pd</span>.</span>concat</span>(</span>[</span>sjJs</span>,</span>pd</span>.</span>Series</span>(</span>True</span>)</span>]</span>)</span>  #</span> 令最后一个时间作为最后一个用水事件的结束时间</span></span>
13</span> #</span> 创建数据框，并定义用水事件序列</span></span>
14</span> sj</span> =</span> pd</span>.</span>DataFrame</span>(</span>np</span>.</span>arange</span>(</span>1</span>,</span>sum</span>(</span>sjKs</span>)</span>+</span>1</span>)</span>,</span>columns</span> =</span> [</span>"</span>事件序号</span>"</span>]</span>)</span></span>
15</span> sj</span>[</span>"</span>事件起始编号</span>"</span>]</span> =</span> data</span>.</span>index</span>[</span>sjKs</span> ==</span> 1</span>]</span>+</span>1</span>  #</span> 定义用水事件的起始编号</span></span>
16</span> sj</span>[</span>"</span>事件终止编号</span>"</span>]</span> =</span> data</span>.</span>index</span>[</span>sjJs</span> ==</span> 1</span>]</span>+</span>1</span>  #</span> 定义用水事件的终止编号</span></span>
17</span> print</span>(</span>'</span>当阈值为4分钟的时候事件数目为：</span>'</span>,</span>sj</span>.</span>shape</span>[</span>0</span>]</span>)</span></span>
18</span> sj</span>.</span>to_csv</span>(</span>'</span>E:/大三下/数据分析/数据/第十章/sj.csv</span>'</span>,</span>index</span> =</span> False</span>)</span></span>
</span></code></pre>3、确定单次用水事件时长阈值</a></h4>
对某热水器用户的数据，根据不同的阈值划分用水事件，得到相应的事件个数。先统计出各个阈值下的用水事件的个数，再通过阈值寻优的方式找出最优的阈值，具体实现方式如代码所示。</p>
1</span> #</span> 确定单次用水事件时长阈值</span></span>
 2</span> n</span> =</span> 4</span>  #</span> 使用以后四个点的平均斜率</span></span>
 3</span> threshold</span> =</span> pd</span>.</span>Timedelta</span>(</span>minutes</span>=</span>5</span>)</span>  #</span> 专家阈值</span></span>
 4</span> data</span>[</span>'</span>发生时间</span>'</span>]</span> =</span> pd</span>.</span>to_datetime</span>(</span>data</span>[</span>'</span>发生时间</span>'</span>]</span>,</span> format</span>=</span>'</span>%Y%m</span>%d</span>%H%M%S</span>'</span>)</span></span>
 5</span> data</span> =</span> data</span>[</span>data</span>[</span>'</span>水流量</span>'</span>]</span> ></span> 0</span>]</span>  #</span> 只要流量大于0的记录</span></span>
 6</span> #</span> 自定义函数：输入划分时间的时间阈值，得到划分的事件数</span></span>
 7</span> def</span> event_num</span>(</span>ts</span>)</span>:</span></span>
 8</span>     d</span> =</span> data</span>[</span>'</span>发生时间</span>'</span>]</span>.</span>diff</span>(</span>)</span> ></span> ts</span>  #</span> 相邻时间作差分，比较是否大于阈值</span></span>
 9</span>     return</span> d</span>.</span>sum</span>(</span>)</span> +</span> 1</span>  #</span> 这样直接返回事件数</span></span>
10</span> dt</span> =</span> [</span>pd</span>.</span>Timedelta</span>(</span>minutes</span>=</span>i</span>)</span> for</span> i</span> in</span> np</span>.</span>arange</span>(</span>1</span>,</span> 9</span>,</span> 0.25</span>)</span>]</span></span>
11</span> h</span> =</span> pd</span>.</span>DataFrame</span>(</span>dt</span>,</span> columns</span>=</span>[</span>'</span>阈值</span>'</span>]</span>)</span>  #</span> 转换数据框，定义阈值列</span></span>
12</span> h</span>[</span>'</span>事件数</span>'</span>]</span> =</span> h</span>[</span>'</span>阈值</span>'</span>]</span>.</span>apply</span>(</span>event_num</span>)</span>  #</span> 计算每个阈值对应的事件数</span></span>
13</span> h</span>[</span>'</span>斜率</span>'</span>]</span> =</span> h</span>[</span>'</span>事件数</span>'</span>]</span>.</span>diff</span>(</span>)</span>/</span>0.25</span>  #</span> 计算每两个相邻点对应的斜率</span></span>
14</span> h</span>[</span>'</span>斜率指标</span>'</span>]</span>=</span> h</span>[</span>'</span>斜率</span>'</span>]</span>.</span>abs</span>(</span>)</span>.</span>rolling</span>(</span>4</span>)</span>.</span>mean</span>(</span>)</span>  #</span> 往前取n个斜率绝对值平均作为斜率指标</span></span>
15</span> ts</span> =</span> h</span>[</span>'</span>阈值</span>'</span>]</span>[</span>h</span>[</span>'</span>斜率指标</span>'</span>]</span>.</span>idxmin</span>(</span>)</span> -</span> n</span>]</span></span>
16</span> #</span> 用idxmin返回最小值的Index，由于rolling_mean()计算的是前n个斜率的绝对值平均</span></span>
17</span> #</span> 所以结果要进行平移（-n）</span></span>
18</span> if</span> ts</span> ></span> threshold</span>:</span></span>
19</span>     ts</span> =</span> pd</span>.</span>Timedelta</span>(</span>minutes</span>=</span>4</span>)</span></span>
20</span> print</span>(</span>'</span>计算出的单次用水时长的阈值为：</span>'</span>,</span>ts</span>)</span></span></code></pre>4、属性构造</a></h4>
（1）构建用水时长与用水频率属性</p>
 1</span> data</span> =</span> pd</span>.</span>read_csv</span>(</span>'</span>E:/大三下/数据分析/数据/第十章/water_heart.csv</span>'</span>)</span>  #</span> 读取热水器使用数据记录</span></span>
 2</span> sj</span> =</span> pd</span>.</span>read_csv</span>(</span>'</span>E:/大三下/数据分析/数据/第十章/sj.csv</span>'</span>)</span>  #</span> 读取用水事件记录</span></span>
 3</span> #</span> 转换时间格式</span></span>
 4</span> data</span>[</span>"</span>发生时间</span>"</span>]</span> =</span> pd</span>.</span>to_datetime</span>(</span>data</span>[</span>"</span>发生时间</span>"</span>]</span>,</span>format</span>=</span>"</span>%Y%m</span>%d</span>%H%M%S</span>"</span>)</span></span>
 5</span> </span></span>
 6</span> #</span> 构造特征：总用水时长</span></span>
 7</span> timeDel</span> =</span> pd</span>.</span>Timedelta</span>(</span>"</span>0.5 sec</span>"</span>)</span></span>
 8</span> sj</span>[</span>"</span>事件开始时间</span>"</span>]</span> =</span> data</span>.</span>iloc</span>[</span>sj</span>[</span>"</span>事件起始编号</span>"</span>]</span>-</span>1</span>,</span>0</span>]</span>.</span>values</span>-</span> timeDel</span></span>
 9</span> sj</span>[</span>"</span>事件结束时间</span>"</span>]</span> =</span> data</span>.</span>iloc</span>[</span>sj</span>[</span>"</span>事件终止编号</span>"</span>]</span>-</span>1</span>,</span>0</span>]</span>.</span>values</span> +</span> timeDel</span></span>
10</span> sj</span>[</span>'</span>洗浴时间点</span>'</span>]</span> =</span> [</span>i</span>.</span>hour</span> for</span> i</span> in</span> sj</span>[</span>"</span>事件开始时间</span>"</span>]</span>]</span></span>
11</span> sj</span>[</span>"</span>总用水时长</span>"</span>]</span> =</span> np</span>.</span>int64</span>(</span>sj</span>[</span>"</span>事件结束时间</span>"</span>]</span> -</span> sj</span>[</span>"</span>事件开始时间</span>"</span>]</span>)</span>/</span>1000000000</span> +</span> 1</span></span>
12</span> </span></span>
13</span> #</span> 构造用水停顿事件</span></span>
14</span> #</span> 构造特征“停顿开始时间”、“停顿结束时间”</span></span>
15</span> #</span> 停顿开始时间指从有水流到无水流，停顿结束时间指从无水流到有水流</span></span>
16</span> for</span> i</span> in</span> range</span>(</span>len</span>(</span>data</span>)</span>-</span>1</span>)</span>:</span></span>
17</span>     if</span> (</span>data</span>.</span>loc</span>[</span>i</span>,</span>"</span>水流量</span>"</span>]</span> !=</span> 0</span>)</span> &</span> (</span>data</span>.</span>loc</span>[</span>i</span> +</span> 1</span>,</span>"</span>水流量</span>"</span>]</span> ==</span> 0</span>)</span> :</span></span>
18</span>         data</span>.</span>loc</span>[</span>i</span> +</span> 1</span>,</span>"</span>停顿开始时间</span>"</span>]</span> =</span> data</span>.</span>loc</span>[</span>i</span> +</span>1</span>,</span> "</span>发生时间</span>"</span>]</span> -</span> timeDel</span></span>
19</span>     if</span> (</span>data</span>.</span>loc</span>[</span>i</span>,</span>"</span>水流量</span>"</span>]</span> ==</span> 0</span>)</span> &</span> (</span>data</span>.</span>loc</span>[</span>i</span> +</span> 1</span>,</span>"</span>水流量</span>"</span>]</span> !=</span> 0</span>)</span> :</span></span>
20</span>         data</span>.</span>loc</span>[</span>i</span>,</span>"</span>停顿结束时间</span>"</span>]</span> =</span> data</span>.</span>loc</span>[</span>i</span> ,</span> "</span>发生时间</span>"</span>]</span> +</span> timeDel</span></span>
21</span>         </span></span>
22</span> #</span> 提取停顿开始时间与结束时间所对应行号，放在数据框Stop中</span></span>
23</span> indStopStart</span> =</span> data</span>.</span>index</span>[</span>data</span>[</span>"</span>停顿开始时间</span>"</span>]</span>.</span>notnull</span>(</span>)</span>]</span>+</span>1</span></span>
24</span> indStopEnd</span> =</span> data</span>.</span>index</span>[</span>data</span>[</span>"</span>停顿结束时间</span>"</span>]</span>.</span>notnull</span>(</span>)</span>]</span>+</span>1</span></span>
25</span> Stop</span> =</span> pd</span>.</span>DataFrame</span>(</span>data</span>=</span>{</span>"</span>停顿开始编号</span>"</span>:</span>indStopStart</span>[</span>:</span>-</span>1</span>]</span>,</span></span>
26</span>                             "</span>停顿结束编号</span>"</span>:</span>indStopEnd</span>[</span>1</span>:</span>]</span>}</span>)</span> </span></span>
27</span> #</span> 计算停顿时长，并放在数据框stop中，停顿时长=停顿结束时间-停顿结束时间</span></span>
28</span> Stop</span>[</span>"</span>停顿时长</span>"</span>]</span> =</span> np</span>.</span>int64</span>(</span>data</span>.</span>loc</span>[</span>indStopEnd</span>[</span>1</span>:</span>]</span>-</span>1</span>,</span>"</span>停顿结束时间</span>"</span>]</span>.</span>values</span>-</span></span>
29</span>                      data</span>.</span>loc</span>[</span>indStopStart</span>[</span>:</span>-</span>1</span>]</span>-</span>1</span>,</span>"</span>停顿开始时间</span>"</span>]</span>.</span>values</span>)</span>/</span>1000000000</span></span>
30</span> #</span> 将每次停顿与事件匹配,停顿的开始时间要大于事件的开始时间，</span></span>
31</span> #</span> 且停顿的结束时间要小于事件的结束时间</span></span>
32</span> for</span> i</span> in</span> range</span>(</span>len</span>(</span>sj</span>)</span>)</span>:</span></span>
33</span>     Stop</span>.</span>loc</span>[</span>(</span>Stop</span>[</span>"</span>停顿开始编号</span>"</span>]</span> ></span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>事件起始编号</span>"</span>]</span>)</span> &</span> </span></span>
34</span>            (</span>Stop</span>[</span>"</span>停顿结束编号</span>"</span>]</span> <</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>事件终止编号</span>"</span>]</span>)</span>,</span>"</span>停顿归属事件</span>"</span>]</span>=</span>i</span>+</span>1</span></span>
35</span>              </span></span>
36</span> #</span> 删除停顿次数为0的事件</span></span>
37</span> Stop</span> =</span> Stop</span>[</span>Stop</span>[</span>"</span>停顿归属事件</span>"</span>]</span>.</span>notnull</span>(</span>)</span>]</span></span>
38</span> </span></span>
39</span> #</span> 构造特征 用水事件停顿总时长、停顿次数、停顿平均时长、</span></span>
40</span> #</span> 用水时长，用水/总时长</span></span>
41</span> stopAgg</span> =</span>  Stop</span>.</span>groupby</span>(</span>"</span>停顿归属事件</span>"</span>)</span>.</span>agg</span>(</span>{</span>"</span>停顿时长</span>"</span>:</span>sum</span>,</span>"</span>停顿开始编号</span>"</span>:</span>len</span>}</span>)</span></span>
42</span> sj</span>.</span>loc</span>[</span>stopAgg</span>.</span>index</span> -</span> 1</span>,</span>"</span>总停顿时长</span>"</span>]</span> =</span> stopAgg</span>.</span>loc</span>[</span>:</span>,</span>"</span>停顿时长</span>"</span>]</span>.</span>values</span></span>
43</span> sj</span>.</span>loc</span>[</span>stopAgg</span>.</span>index</span>-</span>1</span>,</span>"</span>停顿次数</span>"</span>]</span> =</span> stopAgg</span>.</span>loc</span>[</span>:</span>,</span>"</span>停顿开始编号</span>"</span>]</span>.</span>values</span></span>
44</span> sj</span>.</span>fillna</span>(</span>0</span>,</span>inplace</span>=</span>True</span>)</span>  #</span> 对缺失值用0插补</span></span>
45</span> stopNo0</span> =</span> sj</span>[</span>"</span>停顿次数</span>"</span>]</span> !=</span> 0</span>  #</span> 判断用水事件是否存在停顿</span></span>
46</span> sj</span>.</span>loc</span>[</span>stopNo0</span>,</span>"</span>平均停顿时长</span>"</span>]</span> =</span> sj</span>.</span>loc</span>[</span>stopNo0</span>,</span>"</span>总停顿时长</span>"</span>]</span>/</span>sj</span>.</span>loc</span>[</span>stopNo0</span>,</span>"</span>停顿次数</span>"</span>]</span> </span></span>
47</span> sj</span>.</span>fillna</span>(</span>0</span>,</span>inplace</span>=</span>True</span>)</span>  #</span> 对缺失值用0插补</span></span>
48</span> sj</span>[</span>"</span>用水时长</span>"</span>]</span> =</span> sj</span>[</span>"</span>总用水时长</span>"</span>]</span> -</span> sj</span>[</span>"</span>总停顿时长</span>"</span>]</span>  #</span> 定义特征用水时长</span></span>
49</span> sj</span>[</span>"</span>用水/总时长</span>"</span>]</span> =</span> sj</span>[</span>"</span>用水时长</span>"</span>]</span> /</span> sj</span>[</span>"</span>总用水时长</span>"</span>]</span>  #</span> 定义特征 用水/总时长</span></span>
50</span> print</span>(</span>'</span>用水事件用水时长与频率特征构造完成后数据的特征为：</span>\n</span>'</span>,</span>sj</span>.</span>columns</span>)</span></span>
51</span> print</span>(</span>'</span>用水事件用水时长与频率特征构造完成后数据的前5行5列特征为：</span>\n</span>'</span>,</span></span>
52</span>       sj</span>.</span>iloc</span>[</span>:</span>5</span>,</span>:</span>5</span>]</span>)</span></span></code></pre>（2）构建用水量与用水波动属性</a></h4>
除了用水时长、停顿和频率外，用水量也是识别该事件是否为洗浴事件的重要属性。同时用水波动也是区分不同用水事件的关键。</p>
 1</span> #</span> ´代码10-6</span></span>
 2</span> </span></span>
 3</span> data</span>[</span>"</span>水流量</span>"</span>]</span> =</span> data</span>[</span>"</span>水流量</span>"</span>]</span> /</span> 60</span> #</span> 原单位L/min，现转换为L/sec</span></span>
 4</span> sj</span>[</span>"</span>总用水量</span>"</span>]</span> =</span> 0</span> #</span> 给总用水量赋一个初始值0</span></span>
 5</span> for</span> i</span> in</span> range</span>(</span>len</span>(</span>sj</span>)</span>)</span>:</span></span>
 6</span>     Start</span> =</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>事件起始编号</span>"</span>]</span>-</span>1</span></span>
 7</span>     End</span> =</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>事件终止编号</span>"</span>]</span>-</span>1</span></span>
 8</span>     if</span> Start</span> !=</span> End</span>:</span></span>
 9</span>         for</span> j</span> in</span> range</span>(</span>Start</span>,</span>End</span>)</span>:</span></span>
10</span>             if</span> data</span>.</span>loc</span>[</span>j</span>,</span>"</span>水流量</span>"</span>]</span> !=</span> 0</span>:</span></span>
11</span>                 sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>总用水量</span>"</span>]</span> =</span> (</span>data</span>.</span>loc</span>[</span>j</span> +</span> 1</span>,</span>"</span>发生时间</span>"</span>]</span> -</span> </span></span>
12</span>                                     data</span>.</span>loc</span>[</span>j</span>,</span>"</span>发生时间</span>"</span>]</span>)</span>.</span>seconds</span>*</span> \</span></span>
13</span>                                     data</span>.</span>loc</span>[</span>j</span>,</span>"</span>水流量</span>"</span>]</span> +</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>总用水量</span>"</span>]</span></span>
14</span>         sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>总用水量</span>"</span>]</span> =</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>总用水量</span>"</span>]</span> +</span> data</span>.</span>loc</span>[</span>End</span>,</span>"</span>水流量</span>"</span>]</span> *</span> 2</span></span>
15</span>     else</span>:</span></span>
16</span>         sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>总用水量</span>"</span>]</span> =</span> data</span>.</span>loc</span>[</span>Start</span>,</span>"</span>水流量</span>"</span>]</span> *</span> 2</span></span>
17</span>         </span></span>
18</span> sj</span>[</span>"</span>平均水流量</span>"</span>]</span> =</span> sj</span>[</span>"</span>总用水量</span>"</span>]</span> /</span> sj</span>[</span>"</span>用水时长</span>"</span>]</span> #</span> 定义特征 平均水流量</span></span>
19</span> #</span> 构造特征：水流量波动</span></span>
20</span> #</span> 水流量波动=∑(((单次水流的值-平均水流量)^2)*持续时间)/用水时长</span></span>
21</span> sj</span>[</span>"</span>水流量波动</span>"</span>]</span> =</span> 0</span> #</span> 给水流量波动赋一个初始值0</span></span>
22</span> for</span> i</span> in</span> range</span>(</span>len</span>(</span>sj</span>)</span>)</span>:</span></span>
23</span>     Start</span> =</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>事件起始编号</span>"</span>]</span> -</span> 1</span></span>
24</span>     End</span> =</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>事件终止编号</span>"</span>]</span> -</span> 1</span></span>
25</span>     for</span> j</span> in</span> range</span>(</span>Start</span>,</span>End</span> +</span> 1</span>)</span>:</span></span>
26</span>         if</span> data</span>.</span>loc</span>[</span>j</span>,</span>"</span>水流量</span>"</span>]</span> !=</span> 0</span>:</span></span>
27</span>             slbd</span> =</span> (</span>data</span>.</span>loc</span>[</span>j</span>,</span>"</span>水流量</span>"</span>]</span> -</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>平均水流量</span>"</span>]</span>)</span>**</span>2</span></span>
28</span>             slsj</span> =</span> (</span>data</span>.</span>loc</span>[</span>j</span> +</span> 1</span>,</span>"</span>发生时间</span>"</span>]</span> -</span> data</span>.</span>loc</span>[</span>j</span>,</span>"</span>发生时间</span>"</span>]</span>)</span>.</span>seconds</span></span>
29</span>             sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>水流量波动</span>"</span>]</span> =</span> slbd</span> *</span> slsj</span> +</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>水流量波动</span>"</span>]</span></span>
30</span>     sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>水流量波动</span>"</span>]</span> =</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>水流量波动</span>"</span>]</span> /</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>用水时长</span>"</span>]</span>   </span></span>
31</span> </span></span>
32</span> #</span> 构造特征：停顿时长波动</span></span>
33</span> #</span> 停顿时长波动=∑(((单次停顿时长-平均停顿时长)^2)*持续时间)/总停顿时长</span></span>
34</span> sj</span>[</span>"</span>停顿时长波动</span>"</span>]</span> =</span> 0</span> #</span> 给停顿时长波动赋一个初始值0</span></span>
35</span> for</span> i</span> in</span> range</span>(</span>len</span>(</span>sj</span>)</span>)</span>:</span></span>
36</span>     if</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>停顿次数</span>"</span>]</span> ></span> 1</span>:</span> #</span> 当停顿次数为0或1时，停顿时长波动值为0，故排除</span></span>
37</span>         for</span> j</span> in</span> Stop</span>.</span>loc</span>[</span>Stop</span>[</span>"</span>停顿归属事件</span>"</span>]</span> ==</span> (</span>i</span>+</span>1</span>)</span>,</span>"</span>停顿时长</span>"</span>]</span>.</span>values</span>:</span></span>
38</span>             sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>停顿时长波动</span>"</span>]</span> =</span> (</span>(</span>j</span> -</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>平均停顿时长</span>"</span>]</span>)</span>**</span>2</span>)</span> *</span> j</span> +</span> \</span></span>
39</span>                                      sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>停顿时长波动</span>"</span>]</span></span>
40</span>         sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>停顿时长波动</span>"</span>]</span> =</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>停顿时长波动</span>"</span>]</span> /</span> sj</span>.</span>loc</span>[</span>i</span>,</span>"</span>总停顿时长</span>"</span>]</span></span>
41</span> </span></span>
42</span> print</span>(</span>'</span>用水量和波动特征构造完成后数据的特征为：</span>\n</span>'</span>,</span>sj</span>.</span>columns</span>)</span></span>
43</span> print</span>(</span>'</span>用水量和波动特征构造完成后数据的前5行5列特征为：</span>\n</span>'</span>,</span>sj</span>.</span>iloc</span>[</span>:</span>5</span>,</span>:</span>5</span>]</span>)</span></span></code></pre>5、筛选候选洗浴事件</a></h4>
洗浴事件的识别是建立在一次用水事件识别的基础上的，也就是从已经划分好的一次用水事件中识别出哪些一次用水事件是洗浴事件。</p>
可以使用3个比较宽松的条件筛选掉那些非常短暂的用水事件，确定不可能为洗浴事件的数据就删除，剩余的事件称为“候选洗浴事件”。 这3个条件是“或”的关系，也就是说，只要一次完整的用水事件满足任意一个条件，就被判定为短暂用水事件，即会被筛选掉。3个筛选条件如下:</p>
1)一次用水事件中总用水量小于5升。 2)用水时长小于100秒。3)总用水时长小于120秒。</p>
基于构建的用水时长、用水量属性，筛选候选洗浴事件，如以下代码所示。</p>
1</span> #</span> 代码10-7</span></span>
2</span> </span></span>
3</span> sj_bool</span> =</span> (</span>sj</span>[</span>'</span>用水时长</span>'</span>]</span> ></span>100</span>)</span> &</span> (</span>sj</span>[</span>'</span>总用水时长</span>'</span>]</span> ></span> 120</span>)</span> &</span> (</span>sj</span>[</span>'</span>总用水量</span>'</span>]</span> ></span> 5</span>)</span></span>
4</span> sj_final</span> =</span> sj</span>.</span>loc</span>[</span>sj_bool</span>,</span>:</span>]</span></span>
5</span> sj_final</span>.</span>to_excel</span>(</span>'</span>E:/大三下/数据分析/数据/第十章/sj_final.xlsx</span>'</span>,</span>index</span>=</span>False</span>)</span></span>
6</span> print</span>(</span>'</span>筛选出候选洗浴事件前的数据形状为：</span>'</span>,</span>sj</span>.</span>shape</span>)</span></span>
7</span> print</span>(</span>'</span>筛选出候选洗浴事件后的数据形状为：</span>'</span>,</span>sj_final</span>.</span>shape</span>)</span></span></code></pre>
筛选前，用水事件数目总共为172个，经过筛选，余下75个用水事件。结合日志，最终用于建模的属性的总数为11个，其基本情况如下表所示。</p>
（三）模型构建</a></h3>
根据建模样本数据建立BP神经网络模型识别洗浴事件。由于洗浴事件与普通用水事件在特征上存在不同，而且这些不同的特征要被体现出来。于是，根据热水器用户提供的用水日志，将其中洗浴事件的数据状态记录作为训练样本训练BP神经网络。然后根据训练好的BP神经网络来检验新采集的数据，具体过程如下图所示。</p>
在训练BP神经网络的时候，选取了“候选洗浴事件”的11个属性作为BP神经网络的输入，分别为:洗浴时间点、总用水时长、总停顿时长、平均停顿时长、停顿次数、用水时长、用水时长/总用水时长、总用水量、平均水流量、水流量波动和停顿时长波动。训练BP神经网络时给定的输出为1与0，其中1代表该次事件为洗浴事件，0表示该次事件不是洗浴事件。是否为洗浴事件的标签的依据是热水器的用水记录日志。</p>
构建BP神经网络模型需要注意数据本身属性之间存在的量级差异，因此需要进行标准化，消除量级差异。另外，为了便于后续应用模型，可以用joblib.dump函数保存模型。</p>
1</span> import</span> pandas</span> as</span> pd</span></span>
 2</span> from</span> sklearn</span>.</span>preprocessing</span> import</span> StandardScaler</span></span>
 3</span> from</span> sklearn</span>.</span>neural_network</span> import</span> MLPClassifier</span></span>
 4</span> #</span>from sklearn.externals import joblib</span></span>
 5</span> import</span> joblib</span></span>
 6</span> </span></span>
 7</span> #</span> 读取数据</span></span>
 8</span> Xtrain</span> =</span> pd</span>.</span>read_excel</span>(</span>'</span>E:/大三下/数据分析/数据/第十章/sj_final.xlsx</span>'</span>)</span></span>
 9</span> ytrain</span> =</span> pd</span>.</span>read_excel</span>(</span>'</span>E:/大三下/数据分析/数据/第十章/water_heater_log.xlsx</span>'</span>)</span></span>
10</span> test</span> =</span> pd</span>.</span>read_excel</span>(</span>'</span>E:/大三下/数据分析/数据/第十章/test_data.xlsx</span>'</span>)</span></span>
11</span> #</span> 训练集测试集区分。</span></span>
12</span> x_train</span>,</span> x_test</span>,</span> y_train</span>,</span> y_test</span> =</span> Xtrain</span>.</span>iloc</span>[</span>:</span>,</span>5</span>:</span>]</span>,</span>test</span>.</span>iloc</span>[</span>:</span>,</span>4</span>:</span>-</span>1</span>]</span>,</span>\</span></span>
13</span>                                    ytrain</span>.</span>iloc</span>[</span>:</span>,</span>-</span>1</span>]</span>,</span>test</span>.</span>iloc</span>[</span>:</span>,</span>-</span>1</span>]</span></span>
14</span> #</span> 标准化</span></span>
15</span> stdScaler</span> =</span> StandardScaler</span>(</span>)</span>.</span>fit</span>(</span>x_train</span>)</span></span>
16</span> x_stdtrain</span> =</span> stdScaler</span>.</span>transform</span>(</span>x_train</span>)</span></span>
17</span> x_stdtest</span> =</span> stdScaler</span>.</span>transform</span>(</span>x_test</span>)</span></span>
18</span> #</span> 建立模型</span></span>
19</span> bpnn</span> =</span> MLPClassifier</span>(</span>hidden_layer_sizes</span> =</span> (</span>17</span>,</span>10</span>)</span>,</span> max_iter</span> =</span> 200</span>,</span> solver</span> =</span> '</span>lbfgs</span>'</span>,</span>random_state</span>=</span>50</span>)</span></span>
20</span> bpnn</span>.</span>fit</span>(</span>x_stdtrain</span>,</span> y_train</span>)</span></span>
21</span> #</span> 保存模型</span></span>
22</span> joblib</span>.</span>dump</span>(</span>bpnn</span>,</span>'</span>E:/大三下/数据分析/程序/第十章/water_heater_nnet.m</span>'</span>)</span></span>
23</span> print</span>(</span>'</span>构建的模型为：</span>\n</span>'</span>,</span>bpnn</span>)</span></span></code></pre>
在训练BP神经网络时，对神经网络的参数进行了寻优，发现含有两个隐层的神经网络训练效果较好，其中两个隐层的隐节点数分别为17 和10时训练的效果较好。</p>
根据样本，得到训练好的神经网络后，就可以用来识别对应的热水器用户的洗浴事件，其中待检测的样本的11个属性作为输入，输出层输出一个在[-1，1]范围内的值，如果该值小于0，则该事件不是洗浴事件，如果该值大于0，则该事件是洗浴事件。某热水器用户记录了两周的热水器用水日志，将前一周的数据作为训练数据，将后一周的数据作为测试数据，代入上述模型进行测试。</p>
（四）模型检验</a></h3>
结合模型评价的相关知识，使用精确率(precision)、召回率 (recall)和F1值来衡量模型评价的效果较为客观、准确。同时结合ROC曲线，可以更加直观地评价模型的效果。</p>
1</span> #</span> 模型评价</span></span>
 2</span> from</span> sklearn</span>.</span>metrics</span> import</span> classification_report</span></span>
 3</span> from</span> sklearn</span>.</span>metrics</span> import</span> roc_curve</span></span>
 4</span> import</span> joblib</span></span>
 5</span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span>
 6</span> </span></span>
 7</span> bpnn</span> =</span> joblib</span>.</span>load</span>(</span>'</span>E:/大三下/数据分析/程序/第十章/water_heater_nnet.m</span>'</span>)</span>  #</span> 加载模型</span></span>
 8</span> y_pred</span> =</span> bpnn</span>.</span>predict</span>(</span>x_stdtest</span>)</span>  #</span> 返回预测结果</span></span>
 9</span> print</span>(</span>'</span>神经网络预测结果评价报告：</span>\n</span>'</span>,</span>classification_report</span>(</span>y_test</span>,</span>y_pred</span>)</span>)</span></span>
10</span> #</span> 绘制roc曲线图</span></span>
11</span> plt</span>.</span>rcParams</span>[</span>'</span>font.sans-serif</span>'</span>]</span> =</span> '</span>SimHei</span>'</span>  #</span> 显示中文</span></span>
12</span> plt</span>.</span>rcParams</span>[</span>'</span>axes.unicode_minus</span>'</span>]</span> =</span> False</span>  #</span> 显示负号</span></span>
13</span> fpr</span>,</span> tpr</span>,</span> thresholds</span> =</span> roc_curve</span>(</span>y_pred</span>,</span>y_test</span>)</span>  #</span> 求出TPR和FPR</span></span>
14</span> plt</span>.</span>figure</span>(</span>figsize</span>=</span>(</span>6</span>,</span>4</span>)</span>)</span>  #</span> 创建画布</span></span>
15</span> plt</span>.</span>plot</span>(</span>fpr</span>,</span>tpr</span>)</span>  #</span> 绘制曲线</span></span>
16</span> plt</span>.</span>title</span>(</span>'</span>2019320143322 用户用水事件识别ROC曲线</span>'</span>)</span>  #</span> 标题</span></span>
17</span> plt</span>.</span>xlabel</span>(</span>'</span>FPR</span>'</span>)</span>  #</span> x轴标签</span></span>
18</span> plt</span>.</span>ylabel</span>(</span>'</span>TPR</span>'</span>)</span>  #</span> y轴标签</span></span>
19</span> plt</span>.</span>savefig</span>(</span>'</span>E:/大三下/数据分析/数据/第十章/用户用水事件识别ROC曲线.png</span>'</span>)</span>  #</span> 保存图片</span></span>
20</span> plt</span>.</span>show</span>(</span>)</span>  #</span> 显示图形</span></span></code></pre>
根据模型评估报告可以看出，在洗浴事件的识别上精确率 (precision)非常高，达到了90%，同时召回率(recall)也达到了70%以上。综合上述结果，可以确定此次创建的模型是有效且效果良好的，能够用于实际的洗浴事件的识别中。</p>


数据挖掘作业4
Unknown — Sat, 22 Apr 2023 10:09:00 +0800
商品零售购物篮分析</a></h1>
一、背景与挖掘目标</a></h2>
购物篮分析是商业领域最前沿、最具挑战性的问题之一，也是许多企业重点研究的问题。购物篮分析是通过发现顾客在一次购买行为中放入购物篮中不同商品之间的关联，研究顾客的购买行为，从而辅助零售企业制定营销策略的一种数据分析方法。本章使用Apriori关联规则算法实现购物篮分析，发现超市不同商品之间的关联关系，并根据商品之间的关联规则制定销售策略。</p>
通过对商场销售数据进行分析，得到顾客的购买行为特征，并根据发现的规律而采取有效的行动，制定商品摆放、商品定价、新商品采购计划，对增加销量并获取最大利润有重要意义。</p>
根据提供的数据实现以下目标:（1)构建零售商品的Apriori关联规则模型，分析商品之间的关联 性。（2)根据模型结果给出销售策略。</p>
二、分析方法与过程</a></h2>
购物篮关联规则挖掘的主要步骤如下:</p>

对原始数据进行数据探索性分析，分析商品的热销情况与商品结构。</li>
对原始数据进行数据预处理，转换数据形式，使之符合Apriori 关联规则算法要求。</li>
在步骤2得到的建模数据基础上，采用Apriori关联规则算法调整模型输入参数，完成商品关联性分析。</li>
结合实际业务，对模型结果进行分析，根据分析结果给出销售建议，最后输出关联规则结果。</li>
</ol>
###数据探索分析
某商品零售企业共收集了9835个购物篮数据，它主要包括3个属 性:id、Goods和Types。</p>
1、数据特征</a></h3>
某商品零售企业共收集了9835个购物篮数据，它主要包括3个属 性:id、Goods和Types。</p>
#</span> 代码8-1 查看数据特征</span></span>
 2</span> </span></span>
 3</span> import</span> numpy</span> as</span> np</span></span>
 4</span> import</span> pandas</span> as</span> pd</span></span>
 5</span> </span></span>
 6</span> inputfile</span> =</span> '</span>E:/大三下/数据分析/数据/第八章/GoodsOrder.csv</span>'</span>   #</span> 输入的数据文件</span></span>
 7</span> data</span> =</span> pd</span>.</span>read_csv</span>(</span>inputfile</span>,</span>encoding</span> =</span> '</span>gbk</span>'</span>)</span>  #</span> 读取数据</span></span>
 8</span> data</span> .</span>info</span>(</span>)</span>  #</span> 查看数据属性</span></span>
 9</span> </span></span>
10</span> data</span> =</span> data</span>[</span>'</span>id</span>'</span>]</span></span>
11</span> description</span> =</span> [</span>data</span>.</span>count</span>(</span>)</span>,</span>data</span>.</span>min</span>(</span>)</span>,</span> data</span>.</span>max</span>(</span>)</span>]</span>  #</span> 依次计算总数、最小值、最大值</span></span>
12</span> description</span> =</span> pd</span>.</span>DataFrame</span>(</span>description</span>,</span> index</span> =</span> [</span>'</span>Count</span>'</span>,</span>'</span>Min</span>'</span>,</span> '</span>Max</span>'</span>]</span>)</span>.</span>T</span>  #</span> 将结果存入数据框</span></span>
13</span> print</span>(</span>'</span>描述性统计结果：</span>\n</span>'</span>,</span>np</span>.</span>round</span>(</span>description</span>)</span>)</span>  #</span> 输出结果</span></span></code></pre>2、分析热销商品</a></h3>
商品热销情况分析是商品管理中不可或缺的一部分，热销情况分析可以助力商品优选。计算销量排行前10的商品销量及占比，并绘制条形图显示销量前10的商品销量情况。</p>
 1</span> #</span> 代码8-2 分析热销商品</span></span>
 2</span> </span></span>
 3</span> #</span> 销量排行前10商品的销量及其占比</span></span>
 4</span> import</span> pandas</span> as</span> pd</span></span>
 5</span> inputfile</span> =</span> '</span>E:/大三下/数据分析/数据/第八章/GoodsOrder.csv</span>'</span>  #</span> 输入的数据文件</span></span>
 6</span> data</span> =</span> pd</span>.</span>read_csv</span>(</span>inputfile</span>,</span>encoding</span> =</span> '</span>gbk</span>'</span>)</span>  #</span> 读取数据</span></span>
 7</span> group</span> =</span> data</span>.</span>groupby</span>(</span>[</span>'</span>Goods</span>'</span>]</span>)</span>.</span>count</span>(</span>)</span>.</span>reset_index</span>(</span>)</span>  #</span> 对商品进行分类汇总</span></span>
 8</span> sorted</span>=</span>group</span>.</span>sort_values</span>(</span>'</span>id</span>'</span>,</span>ascending</span>=</span>False</span>)</span></span>
 9</span> print</span>(</span>'</span>销量排行前10商品的销量:</span>\n</span>'</span>,</span> sorted</span>[</span>:</span>10</span>]</span>)</span>  #</span> 排序并查看前10位热销商品</span></span>
10</span> </span></span>
11</span> #</span> 画条形图展示出销量排行前10商品的销量</span></span>
12</span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span>
13</span> x</span>=</span>sorted</span>[</span>:</span>10</span>]</span>[</span>'</span>Goods</span>'</span>]</span></span>
14</span> y</span>=</span>sorted</span>[</span>:</span>10</span>]</span>[</span>'</span>id</span>'</span>]</span></span>
15</span> plt</span>.</span>figure</span>(</span>figsize</span> =</span> (</span>8</span>,</span> 4</span>)</span>)</span>  #</span> 设置画布大小 </span></span>
16</span> plt</span>.</span>barh</span>(</span>x</span>,</span>y</span>)</span></span>
17</span> plt</span>.</span>rcParams</span>[</span>'</span>font.sans-serif</span>'</span>]</span> =</span> '</span>SimHei</span>'</span></span>
18</span> plt</span>.</span>xlabel</span>(</span>'</span>销量</span>'</span>)</span>  #</span> 设置x轴标题</span></span>
19</span> plt</span>.</span>ylabel</span>(</span>'</span>商品类别</span>'</span>)</span>  #</span> 设置y轴标题</span></span>
20</span> plt</span>.</span>title</span>(</span>'</span>2019320143322 商品的销量TOP10</span>'</span>)</span>  #</span> 设置标题</span></span>
21</span> plt</span>.</span>savefig</span>(</span>'</span>E:/大三下/数据分析/数据/第八章/top10.png</span>'</span>)</span>  #</span> 把图片以.png格式保存</span></span>
22</span> plt</span>.</span>show</span>(</span>)</span>  #</span> 展示图片</span></span>
23</span> </span></span>
24</span> #</span> 销量排行前10商品的销量占比</span></span>
25</span> data_nums</span> =</span> data</span>.</span>shape</span>[</span>0</span>]</span></span>
26</span> for</span> idnex</span>,</span> row</span> in</span> sorted</span>[</span>:</span>10</span>]</span>.</span>iterrows</span>(</span>)</span>:</span></span>
27</span>     print</span>(</span>row</span>[</span>'</span>Goods</span>'</span>]</span>,</span>row</span>[</span>'</span>id</span>'</span>]</span>,</span>row</span>[</span>'</span>id</span>'</span>]</span>/</span>data_nums</span>)</span></span></code></pre>
通过分析热销商品的结果可知，全脂牛奶的销售量最高，为2513 件，占比5.795%;其次是其他蔬菜、面包卷和苏打，占比分别为 4.388%、4.171%、3.955%。</p>
3、分析商品结构</a></h3>
对每一类商品的热销程度进行分析，有利于商家制定商品在货架上的摆放策略和位置，若是某类商品较为热销，商场可以把此类商品摆放到商场的中心位置，以方便顾客选购;或者是放在商场深处的位置，使顾客在购买热销商品前经过非热销商品所在位置，增加在非热销商品处的停留时间，以促进非热销商品的销量。</p>
原始数据中的商品本身已经经过归类处理，但是部分商品还是存在 一定的重叠，故需要再次对其进行归类处理。分析归类后各类别商品的销量及其占比后，绘制饼图来显示各类商品的销量占比情况。</p>
 1</span> #</span> 代码8-3 各类别商品的销量及其占比</span></span>
 2</span> </span></span>
 3</span> import</span> pandas</span> as</span> pd</span></span>
 4</span> inputfile1</span> =</span> '</span>E:/大三下/数据分析/数据/第八章/GoodsOrder.csv</span>'</span></span>
 5</span> inputfile2</span> =</span> '</span>E:/大三下/数据分析/数据/第八章/GoodsTypes.csv</span>'</span></span>
 6</span> data</span> =</span> pd</span>.</span>read_csv</span>(</span>inputfile1</span>,</span>encoding</span> =</span> '</span>gbk</span>'</span>)</span></span>
 7</span> types</span> =</span> pd</span>.</span>read_csv</span>(</span>inputfile2</span>,</span>encoding</span> =</span> '</span>gbk</span>'</span>)</span>  #</span> 读入数据</span></span>
 8</span> </span></span>
 9</span> group</span> =</span> data</span>.</span>groupby</span>(</span>[</span>'</span>Goods</span>'</span>]</span>)</span>.</span>count</span>(</span>)</span>.</span>reset_index</span>(</span>)</span></span>
10</span> sort</span> =</span> group</span>.</span>sort_values</span>(</span>'</span>id</span>'</span>,</span>ascending</span> =</span> False</span>)</span>.</span>reset_index</span>(</span>)</span></span>
11</span> data_nums</span> =</span> data</span>.</span>shape</span>[</span>0</span>]</span>  #</span> 总量</span></span>
12</span> del</span> sort</span>[</span>'</span>index</span>'</span>]</span></span>
13</span> </span></span>
14</span> sort_links</span> =</span> pd</span>.</span>merge</span>(</span>sort</span>,</span>types</span>)</span>  #</span> 合并两个datafreame 根据type</span></span>
15</span> #</span> 根据类别求和，每个商品类别的总量，并排序</span></span>
16</span> sort_link</span> =</span> sort_links</span>.</span>groupby</span>(</span>[</span>'</span>Types</span>'</span>]</span>)</span>.</span>sum</span>(</span>)</span>.</span>reset_index</span>(</span>)</span></span>
17</span> sort_link</span> =</span> sort_link</span>.</span>sort_values</span>(</span>'</span>id</span>'</span>,</span>ascending</span> =</span> False</span>)</span>.</span>reset_index</span>(</span>)</span></span>
18</span> del</span> sort_link</span>[</span>'</span>index</span>'</span>]</span>  #</span> 删除“index”列</span></span>
19</span> </span></span>
20</span> #</span> 求百分比，然后更换列名，最后输出到文件</span></span>
21</span> sort_link</span>[</span>'</span>count</span>'</span>]</span> =</span> sort_link</span>.</span>apply</span>(</span>lambda</span> line</span>:</span> line</span>[</span>'</span>id</span>'</span>]</span>/</span>data_nums</span>,</span>axis</span>=</span>1</span>)</span></span>
22</span> sort_link</span>.</span>rename</span>(</span>columns</span> =</span> {</span>'</span>count</span>'</span>:</span>'</span>percent</span>'</span>}</span>,</span>inplace</span> =</span> True</span>)</span></span>
23</span> print</span>(</span>'</span>各类别商品的销量及其占比:</span>\n</span>'</span>,</span>sort_link</span>)</span></span>
24</span> outfile1</span> =</span> '</span>E:/大三下/数据分析/数据/第八章/percent.csv</span>'</span></span>
25</span> sort_link</span>.</span>to_csv</span>(</span>outfile1</span>,</span>index</span> =</span> False</span>,</span>header</span> =</span> True</span>,</span>encoding</span>=</span>'</span>gbk</span>'</span>)</span>  #</span> 保存结果</span></span>
26</span> </span></span>
27</span> #</span> 画饼图展示每类商品销量占比</span></span>
28</span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span>
29</span> data</span> =</span> sort_link</span>[</span>'</span>percent</span>'</span>]</span></span>
30</span> labels</span> =</span> sort_link</span>[</span>'</span>Types</span>'</span>]</span></span>
31</span> plt</span>.</span>figure</span>(</span>figsize</span>=</span>(</span>8</span>,</span> 6</span>)</span>)</span>  #</span> 设置画布大小   </span></span>
32</span> plt</span>.</span>pie</span>(</span>data</span>,</span>labels</span>=</span>labels</span>,</span>autopct</span>=</span>'</span>%1.2f</span>%%</span>'</span>)</span></span>
33</span> plt</span>.</span>rcParams</span>[</span>'</span>font.sans-serif</span>'</span>]</span> =</span> '</span>SimHei</span>'</span></span>
34</span> plt</span>.</span>title</span>(</span>'</span>2019320143322 每类商品销量占比</span>'</span>)</span>  #</span> 设置标题</span></span>
35</span> plt</span>.</span>savefig</span>(</span>'</span>E:/大三下/数据分析/数据/第八章/persent.png</span>'</span>)</span>  #</span> 把图片以.png格式保存</span></span>
36</span> plt</span>.</span>show</span>(</span>)</span></span></code></pre>
通过分析各类别商品的销量及其占比情况可知，非酒精饮料、西点、果蔬3类商品的销量差距不大，占总销量的50%左右，同时，根据大类划分发现，和食品类的销量总和接近90%，说明顾客倾向于购买此类商品，而其余商品仅是商场为满足顾客的其他需求而设定的，并非销售的主力军。</p>
（1）非酒精饮料类商品的内部商品结构</a></h4>
进一步查看销量第一的非酒精饮料类商品的内部商品结构，并绘制饼图显示其销量占比情况。</p>
 1</span> #</span> 代码8-4 非酒精饮料内部商品的销量及其占比</span></span>
 2</span> </span></span>
 3</span> #</span> 先筛选“非酒精饮料”类型的商品，然后求百分比，然后输出结果到文件。</span></span>
 4</span> selected</span> =</span> sort_links</span>.</span>loc</span>[</span>sort_links</span>[</span>'</span>Types</span>'</span>]</span> ==</span> '</span>非酒精饮料</span>'</span>]</span>  #</span> 挑选商品类别为“非酒精饮料”并排序</span></span>
 5</span> child_nums</span> =</span> selected</span>[</span>'</span>id</span>'</span>]</span>.</span>sum</span>(</span>)</span>  #</span> 对所有的“非酒精饮料”求和</span></span>
 6</span> selected</span>[</span>'</span>child_percent</span>'</span>]</span> =</span> selected</span>.</span>apply</span>(</span>lambda</span> line</span>:</span> line</span>[</span>'</span>id</span>'</span>]</span>/</span>child_nums</span>,</span>axis</span> =</span> 1</span>)</span>  #</span> 求百分比</span></span>
 7</span> selected</span>.</span>rename</span>(</span>columns</span> =</span> {</span>'</span>id</span>'</span>:</span>'</span>count</span>'</span>}</span>,</span>inplace</span> =</span> True</span>)</span></span>
 8</span> print</span>(</span>'</span>非酒精饮料内部商品的销量及其占比:</span>\n</span>'</span>,</span>selected</span>)</span></span>
 9</span> outfile2</span> =</span> '</span>E:/大三下/数据分析/数据/第八章/child_percent.csv</span>'</span></span>
10</span> sort_link</span>.</span>to_csv</span>(</span>outfile2</span>,</span>index</span> =</span> False</span>,</span>header</span> =</span> True</span>,</span>encoding</span>=</span>'</span>gbk</span>'</span>)</span>  #</span> 输出结果</span></span>
11</span> </span></span>
12</span> #</span> 画饼图展示非酒精饮品内部各商品的销量占比</span></span>
13</span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span>
14</span> data</span> =</span> selected</span>[</span>'</span>child_percent</span>'</span>]</span></span>
15</span> labels</span> =</span> selected</span>[</span>'</span>Goods</span>'</span>]</span></span>
16</span> plt</span>.</span>figure</span>(</span>figsize</span> =</span> (</span>8</span>,</span>6</span>)</span>)</span>  #</span> 设置画布大小 </span></span>
17</span> explode</span> =</span> (</span>0.02</span>,</span>0.03</span>,</span>0.04</span>,</span>0.05</span>,</span>0.06</span>,</span>0.07</span>,</span>0.08</span>,</span>0.08</span>,</span>0.3</span>,</span>0.1</span>,</span>0.3</span>)</span>  #</span> 设置每一块分割出的间隙大小</span></span>
18</span> plt</span>.</span>pie</span>(</span>data</span>,</span>explode</span> =</span> explode</span>,</span>labels</span> =</span> labels</span>,</span>autopct</span> =</span> '</span>%1.2f</span>%%</span>'</span>,</span></span>
19</span>         pctdistance</span> =</span> 1.1</span>,</span>labeldistance</span> =</span> 1.2</span>)</span></span>
20</span> plt</span>.</span>rcParams</span>[</span>'</span>font.sans-serif</span>'</span>]</span> =</span> '</span>SimHei</span>'</span></span>
21</span> plt</span>.</span>title</span>(</span>"</span>2019320143322 非酒精饮料内部各商品的销量占比</span>"</span>)</span>  #</span> 设置标题</span></span>
22</span> plt</span>.</span>axis</span>(</span>'</span>equal</span>'</span>)</span></span>
23</span> plt</span>.</span>savefig</span>(</span>'</span>E:/大三下/数据分析/数据/第八章/child_persent.png</span>'</span>)</span>  #</span> 保存图形</span></span>
24</span> plt</span>.</span>show</span>(</span>)</span>  #</span> 展示图形</span></span></code></pre>
通过分析非酒精饮料内部商品的销量及其占比情况可知，全脂牛奶的销量在非酒精饮料的总销量中占比超过33%，前3种非酒精饮料的销量在非酒精饮料的总销量中的占比接近70%，这就说明大部分顾客到店购买的饮料为这3种，而商场就需要时常注意货物的库存，定期补货。</p>
西点类商品的内部商品结构</a></h4>
1</span> #</span> 西点类的内部商品结构</span></span>
 2</span> </span></span>
 3</span> #</span> 先筛选“西点”类型的商品，然后求百分比，然后输出结果到文件。</span></span>
 4</span> selected</span> =</span> sort_links</span>.</span>loc</span>[</span>sort_links</span>[</span>'</span>Types</span>'</span>]</span> ==</span> '</span>西点</span>'</span>]</span>  #</span> 挑选商品类别为“西点”并排序</span></span>
 5</span> child_nums</span> =</span> selected</span>[</span>'</span>id</span>'</span>]</span>.</span>sum</span>(</span>)</span>  #</span> 对所有的“西点”求和</span></span>
 6</span> selected</span>[</span>'</span>child_percent_xidian</span>'</span>]</span> =</span> selected</span>.</span>apply</span>(</span>lambda</span> line</span>:</span> line</span>[</span>'</span>id</span>'</span>]</span>/</span>child_nums</span>,</span>axis</span> =</span> 1</span>)</span>  #</span> 求百分比</span></span>
 7</span> selected</span>.</span>rename</span>(</span>columns</span> =</span> {</span>'</span>id</span>'</span>:</span>'</span>count</span>'</span>}</span>,</span>inplace</span> =</span> True</span>)</span></span>
 8</span> print</span>(</span>'</span>西点内部商品的销量及其占比:</span>\n</span>'</span>,</span>selected</span>)</span></span>
 9</span> outfile2</span> =</span> '</span>E:/大三下/数据分析/数据/第八章/child_percent_xidian.csv</span>'</span></span>
10</span> sort_link</span>.</span>to_csv</span>(</span>outfile2</span>,</span>index</span> =</span> False</span>,</span>header</span> =</span> True</span>,</span>encoding</span>=</span>'</span>gbk</span>'</span>)</span>  #</span> 输出结果</span></span>
11</span> #</span> 画饼图展示西点内部各商品的销量占比</span></span>
12</span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span>
13</span> data</span> =</span> selected</span>[</span>'</span>child_percent_xidian</span>'</span>]</span></span>
14</span> labels</span> =</span> selected</span>[</span>'</span>Goods</span>'</span>]</span></span>
15</span> plt</span>.</span>figure</span>(</span>figsize</span> =</span> (</span>8</span>,</span>6</span>)</span>)</span>  #</span> 设置画布大小 </span></span>
16</span> explode</span> =</span> (</span>0.05</span>,</span>0.04</span>,</span>0.04</span>,</span>0.05</span>,</span>0.06</span>,</span>0.07</span>,</span>0.03</span>,</span>0.03</span>,</span>0.03</span>,</span>0.02</span>,</span>0.03</span>,</span>0.02</span>,</span>0.02</span>,</span>0.02</span>,</span>0.02</span>,</span>0.08</span>,</span>0.3</span>,</span>0.34</span>,</span>0.38</span>,</span>0.4</span>,</span>0.8</span>)</span>  #</span> 设置每一块分割出的间隙大小</span></span>
17</span> plt</span>.</span>pie</span>(</span>data</span>,</span>explode</span> =</span> explode</span>,</span>labels</span> =</span> labels</span>,</span>autopct</span> =</span> '</span>%1.2f</span>%%</span>'</span>,</span></span>
18</span>         pctdistance</span> =</span> 1.1</span>,</span>labeldistance</span> =</span> 1.2</span>)</span></span>
19</span> plt</span>.</span>rcParams</span>[</span>'</span>font.sans-serif</span>'</span>]</span> =</span> '</span>SimHei</span>'</span></span>
20</span> plt</span>.</span>title</span>(</span>"</span>2019320143322 西点内部各商品的销量占比</span>"</span>)</span>  #</span> 设置标题</span></span>
21</span> plt</span>.</span>axis</span>(</span>'</span>equal</span>'</span>)</span></span>
22</span> plt</span>.</span>savefig</span>(</span>'</span>E:/大三下/数据分析/数据/第八章/child_persent_xidian.png</span>'</span>)</span>  #</span> 保存图形</span></span>
23</span> plt</span>.</span>show</span>(</span>)</span>  #</span> 展示图形</span></span></code></pre>（3）果蔬的内部商品结构</a></h4>
1</span> #</span> 果蔬类的内部商品结构</span></span>
 2</span> </span></span>
 3</span> #</span> 先筛选“果蔬”类型的商品，然后求百分比，然后输出结果到文件。</span></span>
 4</span> selected</span> =</span> sort_links</span>.</span>loc</span>[</span>sort_links</span>[</span>'</span>Types</span>'</span>]</span> ==</span> '</span>果蔬</span>'</span>]</span>  #</span> 挑选商品类别为“果蔬”并排序</span></span>
 5</span> child_nums</span> =</span> selected</span>[</span>'</span>id</span>'</span>]</span>.</span>sum</span>(</span>)</span>  #</span> 对所有的“果蔬”求和</span></span>
 6</span> selected</span>[</span>'</span>child_percent_guoshu</span>'</span>]</span> =</span> selected</span>.</span>apply</span>(</span>lambda</span> line</span>:</span> line</span>[</span>'</span>id</span>'</span>]</span>/</span>child_nums</span>,</span>axis</span> =</span> 1</span>)</span>  #</span> 求百分比</span></span>
 7</span> selected</span>.</span>rename</span>(</span>columns</span> =</span> {</span>'</span>id</span>'</span>:</span>'</span>count</span>'</span>}</span>,</span>inplace</span> =</span> True</span>)</span></span>
 8</span> print</span>(</span>'</span>果蔬内部商品的销量及其占比:</span>\n</span>'</span>,</span>selected</span>)</span></span>
 9</span> outfile2</span> =</span> '</span>E:/大三下/数据分析/数据/第八章/child_percent_guoshu.csv</span>'</span></span>
10</span> sort_link</span>.</span>to_csv</span>(</span>outfile2</span>,</span>index</span> =</span> False</span>,</span>header</span> =</span> True</span>,</span>encoding</span>=</span>'</span>gbk</span>'</span>)</span>  #</span> 输出结果</span></span>
11</span> #</span> 画饼图展示果蔬内部各商品的销量占比</span></span>
12</span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span>
13</span> data</span> =</span> selected</span>[</span>'</span>child_percent_guoshu</span>'</span>]</span></span>
14</span> labels</span> =</span> selected</span>[</span>'</span>Goods</span>'</span>]</span></span>
15</span> plt</span>.</span>figure</span>(</span>figsize</span> =</span> (</span>8</span>,</span>6</span>)</span>)</span>  #</span> 设置画布大小 </span></span>
16</span> explode</span> =</span> (</span>0.02</span>,</span>0.02</span>,</span>0.04</span>,</span>0.03</span>,</span>0.04</span>,</span>0.04</span>,</span>0.03</span>,</span>0.02</span>,</span>0.01</span>,</span>0.03</span>,</span>0.08</span>,</span>0.2</span>,</span>0.4</span>,</span>0.6</span>)</span>  #</span> 设置每一块分割出的间隙大小</span></span>
17</span> plt</span>.</span>pie</span>(</span>data</span>,</span>explode</span> =</span> explode</span>,</span>labels</span> =</span> labels</span>,</span>autopct</span> =</span> '</span>%1.2f</span>%%</span>'</span>,</span></span>
18</span>         pctdistance</span> =</span> 1.1</span>,</span>labeldistance</span> =</span> 1.2</span>)</span></span>
19</span> plt</span>.</span>rcParams</span>[</span>'</span>font.sans-serif</span>'</span>]</span> =</span> '</span>SimHei</span>'</span></span>
20</span> plt</span>.</span>title</span>(</span>"</span>2019320143322 果蔬内部各商品的销量占比</span>"</span>)</span>  #</span> 设置标题</span></span>
21</span> plt</span>.</span>axis</span>(</span>'</span>equal</span>'</span>)</span></span>
22</span> plt</span>.</span>savefig</span>(</span>'</span>E:/大三下/数据分析/数据/第八章/child_persent_guoshu.png</span>'</span>)</span>  #</span> 保存图形</span></span>
23</span> plt</span>.</span>show</span>(</span>)</span>  #</span> 展示图形</span></span></code></pre>（4）米粮调料的内部商品结构</a></h4>
1</span> #</span> 米粮调料的内部商品结构</span></span>
 2</span> </span></span>
 3</span> #</span> 先筛选“米粮调料”类型的商品，然后求百分比，然后输出结果到文件。</span></span>
 4</span> selected</span> =</span> sort_links</span>.</span>loc</span>[</span>sort_links</span>[</span>'</span>Types</span>'</span>]</span> ==</span> '</span>米粮调料</span>'</span>]</span>  #</span> 挑选商品类别为“米粮调料”并排序</span></span>
 5</span> child_nums</span> =</span> selected</span>[</span>'</span>id</span>'</span>]</span>.</span>sum</span>(</span>)</span>  #</span> 对所有的“米粮调料”求和</span></span>
 6</span> selected</span>[</span>'</span>child_percent_miliang</span>'</span>]</span> =</span> selected</span>.</span>apply</span>(</span>lambda</span> line</span>:</span> line</span>[</span>'</span>id</span>'</span>]</span>/</span>child_nums</span>,</span>axis</span> =</span> 1</span>)</span>  #</span> 求百分比</span></span>
 7</span> selected</span>.</span>rename</span>(</span>columns</span> =</span> {</span>'</span>id</span>'</span>:</span>'</span>count</span>'</span>}</span>,</span>inplace</span> =</span> True</span>)</span></span>
 8</span> print</span>(</span>'</span>米粮调料内部商品的销量及其占比:</span>\n</span>'</span>,</span>selected</span>)</span></span>
 9</span> outfile2</span> =</span> '</span>E:/大三下/数据分析/数据/第八章/child_percent_miliang.csv</span>'</span></span>
10</span> sort_link</span>.</span>to_csv</span>(</span>outfile2</span>,</span>index</span> =</span> False</span>,</span>header</span> =</span> True</span>,</span>encoding</span>=</span>'</span>gbk</span>'</span>)</span>  #</span> 输出结果</span></span>
11</span> #</span> 画饼图展示米粮调料内部各商品的销量占比</span></span>
12</span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span>
13</span> data</span> =</span> selected</span>[</span>'</span>child_percent_miliang</span>'</span>]</span></span>
14</span> labels</span> =</span> selected</span>[</span>'</span>Goods</span>'</span>]</span></span>
15</span> plt</span>.</span>figure</span>(</span>figsize</span> =</span> (</span>8</span>,</span>6</span>)</span>)</span>  #</span> 设置画布大小 </span></span>
16</span> explode</span> =</span> (</span>0.05</span>,</span>0.04</span>,</span>0.04</span>,</span>0.05</span>,</span>0.06</span>,</span>0.07</span>,</span>0.03</span>,</span>0.03</span>,</span>0.03</span>,</span>0.02</span>,</span>0.03</span>,</span>0.02</span>,</span>0.02</span>,</span>0.02</span>,</span>0.02</span>,</span>0.02</span>,</span>0.02</span>,</span>0.02</span>,</span>0.02</span>,</span>0.02</span>,</span>0.02</span>,</span>0.02</span>,</span>0.02</span>,</span>0.03</span>,</span>0.03</span>,</span>0.03</span>,</span>0.08</span>,</span>0.3</span>,</span>0.34</span>,</span>0.38</span>,</span>0.4</span>,</span>0.8</span>)</span>  #</span> 设置每一块分割出的间隙大小</span></span>
17</span> plt</span>.</span>pie</span>(</span>data</span>,</span>explode</span> =</span> explode</span>,</span>labels</span> =</span> labels</span>,</span>autopct</span> =</span> '</span>%1.2f</span>%%</span>'</span>,</span></span>
18</span>         pctdistance</span> =</span> 1.1</span>,</span>labeldistance</span> =</span> 1.2</span>)</span></span>
19</span> plt</span>.</span>rcParams</span>[</span>'</span>font.sans-serif</span>'</span>]</span> =</span> '</span>SimHei</span>'</span></span>
20</span> plt</span>.</span>title</span>(</span>"</span>2019320143322 米粮调料内部各商品的销量占比</span>"</span>)</span>  #</span> 设置标题</span></span>
21</span> plt</span>.</span>axis</span>(</span>'</span>equal</span>'</span>)</span></span>
22</span> plt</span>.</span>savefig</span>(</span>'</span>E:/大三下/数据分析/数据/第八章/child_persent_miliang.png</span>'</span>)</span>  #</span> 保存图形</span></span>
23</span> plt</span>.</span>show</span>(</span>)</span>  #</span> 展示图形</span></span></code></pre>（二）数据预处理</a></h3>
通过对数据探索分析发现数据完整，并不存在缺失值。建模之前需要转变数据的格式，才能使用Apriori函数进行关联分析。</p>
</span>
 1</span> #</span> 代码8-5 数据转换</span></span>
 2</span> </span></span>
 3</span> import</span> pandas</span> as</span> pd</span></span>
 4</span> inputfile</span>=</span>'</span>E:/大三下/数据分析/数据/第八章/GoodsOrder.csv</span>'</span></span>
 5</span> data</span> =</span> pd</span>.</span>read_csv</span>(</span>inputfile</span>,</span>encoding</span> =</span> '</span>gbk</span>'</span>)</span></span>
 6</span> </span></span>
 7</span> #</span> 根据id对“Goods”列合并，并使用“，”将各商品隔开</span></span>
 8</span> data</span>[</span>'</span>Goods</span>'</span>]</span> =</span> data</span>[</span>'</span>Goods</span>'</span>]</span>.</span>apply</span>(</span>lambda</span> x</span>:</span>'</span>,</span>'</span>+</span>x</span>)</span></span>
 9</span> data</span> =</span> data</span>.</span>groupby</span>(</span>'</span>id</span>'</span>)</span>.</span>sum</span>(</span>)</span>.</span>reset_index</span>(</span>)</span></span>
10</span> </span></span>
11</span> #</span> 对合并的商品列转换数据格式</span></span>
12</span> data</span>[</span>'</span>Goods</span>'</span>]</span> =</span> data</span>[</span>'</span>Goods</span>'</span>]</span>.</span>apply</span>(</span>lambda</span> x</span> :</span>[</span>x</span>[</span>1</span>:</span>]</span>]</span>)</span></span>
13</span> data_list</span> =</span> list</span>(</span>data</span>[</span>'</span>Goods</span>'</span>]</span>)</span></span>
14</span> </span></span>
15</span> #</span> 分割商品名为每个元素</span></span>
16</span> data_translation</span> =</span> [</span>]</span></span>
17</span> for</span> i</span> in</span> data_list</span>:</span></span>
18</span>     p</span> =</span> i</span>[</span>0</span>]</span>.</span>split</span>(</span>'</span>,</span>'</span>)</span></span>
19</span>     data_translation</span>.</span>append</span>(</span>p</span>)</span></span>
20</span> print</span>(</span>'</span>数据转换结果的前5个元素：</span>\n</span>'</span>,</span> data_translation</span>[</span>0</span>:</span>5</span>]</span>)</span></span></code></pre>（三）模型构建</a></h3>
本案例的目标是探索商品之间的关联关系，因此采用关联规则算法，以挖掘它们之间的关联关系。关联规则算法主要用于寻找数据中项集之间的关联关系，它揭示了数据项间的未知关系。基于样本的统计规 律，进行关联规则分析。根据所分析的关联关系，可通过一个属性的信息来推断另一个属性的信息。当置信度达到某一阈值时，就可以认为规则成立。Apriori算法是常用的关联规则算法之一，也是最为经典的分析频繁项集的算法，它是第一次实现在大数据集上可行的关联规则提取的算法。除此之外，还有FP-Tree算法，Eclat算法和灰色关联算法等。本案例主要使用Apriori算法进行分析。</p>

商品购物篮关联规则模型构建
由图可知，模型主要由输入、算法处理、输出3个部分组成。输 入部分包括建模样本数据的输入和建模参数的输入。算法处理部分是采用Apriori关联规则算法进行处理。输出部分为采用Apriori关联规则算法进行处理后的结果。</li>
</ol>
模型具体实现步骤:首先设置建模参数最小支持度、最小置信度， 输入建模样本数据;然后采用Apriori关联规则算法对建模的样本数据进行分析，以模型参数设置的最小支持度、最小置信度以及分析目标作为条件，如果所有的规则都不满足条件，则需要重新调整模型参数，否则输出关联规则结果。</p>
目前，如何设置最小支持度与最小置信度并没有统一的标准。大部分都是根据业务经验设置初始值，然后经过多次调整，获取与业务相符的关联规则结果。本案例经过多次调整并结合实际业务分析，选取模型的输入参数为:最小支持度0.02、最小置信度0.35。其关联规则代码如下。</p>
</span>
1</span> #</span> 代码8-6 构建关联规则模型</span></span>
  2</span> </span></span>
  3</span> from</span> numpy</span> import</span> *</span></span>
  4</span>  </span></span>
  5</span> def</span> loadDataSet</span>(</span>)</span>:</span></span>
  6</span>     return</span> [</span>[</span>'</span>a</span>'</span>,</span> '</span>c</span>'</span>,</span> '</span>e</span>'</span>]</span>,</span> [</span>'</span>b</span>'</span>,</span> '</span>d</span>'</span>]</span>,</span> [</span>'</span>b</span>'</span>,</span> '</span>c</span>'</span>]</span>,</span> [</span>'</span>a</span>'</span>,</span> '</span>b</span>'</span>,</span> '</span>c</span>'</span>,</span> '</span>d</span>'</span>]</span>,</span> [</span>'</span>a</span>'</span>,</span> '</span>b</span>'</span>]</span>,</span> [</span>'</span>b</span>'</span>,</span> '</span>c</span>'</span>]</span>,</span> [</span>'</span>a</span>'</span>,</span> '</span>b</span>'</span>]</span>,</span></span>
  7</span>             [</span>'</span>a</span>'</span>,</span> '</span>b</span>'</span>,</span> '</span>c</span>'</span>,</span> '</span>e</span>'</span>]</span>,</span> [</span>'</span>a</span>'</span>,</span> '</span>b</span>'</span>,</span> '</span>c</span>'</span>]</span>,</span> [</span>'</span>a</span>'</span>,</span> '</span>c</span>'</span>,</span> '</span>e</span>'</span>]</span>]</span></span>
  8</span>  </span></span>
  9</span> def</span> createC1</span>(</span>dataSet</span>)</span>:</span></span>
 10</span>     C1</span> =</span> [</span>]</span></span>
 11</span>     for</span> transaction</span> in</span> dataSet</span>:</span></span>
 12</span>         for</span> item</span> in</span> transaction</span>:</span></span>
 13</span>             if</span> not</span> [</span>item</span>]</span> in</span> C1</span>:</span></span>
 14</span>                 C1</span>.</span>append</span>(</span>[</span>item</span>]</span>)</span></span>
 15</span>     C1</span>.</span>sort</span>(</span>)</span></span>
 16</span>     #</span> 映射为frozenset唯一性的，可使用其构造字典</span></span>
 17</span>     return</span> list</span>(</span>map</span>(</span>frozenset</span>,</span> C1</span>)</span>)</span>     </span></span>
 18</span>     </span></span>
 19</span> #</span> 从候选K项集到频繁K项集（支持度计算）</span></span>
 20</span> def</span> scanD</span>(</span>D</span>,</span> Ck</span>,</span> minSupport</span>)</span>:</span></span>
 21</span>     ssCnt</span> =</span> {</span>}</span></span>
 22</span>     for</span> tid</span> in</span> D</span>:</span>   #</span> 遍历数据集</span></span>
 23</span>         for</span> can</span> in</span> Ck</span>:</span>  #</span> 遍历候选项</span></span>
 24</span>             if</span> can</span>.</span>issubset</span>(</span>tid</span>)</span>:</span>  #</span> 判断候选项中是否含数据集的各项</span></span>
 25</span>                 if</span> not</span> can</span> in</span> ssCnt</span>:</span></span>
 26</span>                     ssCnt</span>[</span>can</span>]</span> =</span> 1</span>  #</span> 不含设为1</span></span>
 27</span>                 else</span>:</span></span>
 28</span>                     ssCnt</span>[</span>can</span>]</span> +=</span> 1</span>  #</span> 有则计数加1</span></span>
 29</span>     numItems</span> =</span> float</span>(</span>len</span>(</span>D</span>)</span>)</span>  #</span> 数据集大小</span></span>
 30</span>     retList</span> =</span> [</span>]</span>  #</span> L1初始化</span></span>
 31</span>     supportData</span> =</span> {</span>}</span>  #</span> 记录候选项中各个数据的支持度</span></span>
 32</span>     for</span> key</span> in</span> ssCnt</span>:</span></span>
 33</span>         support</span> =</span> ssCnt</span>[</span>key</span>]</span> /</span> numItems</span>  #</span> 计算支持度</span></span>
 34</span>         if</span> support</span> >=</span> minSupport</span>:</span></span>
 35</span>             retList</span>.</span>insert</span>(</span>0</span>,</span> key</span>)</span>  #</span> 满足条件加入L1中</span></span>
 36</span>             supportData</span>[</span>key</span>]</span> =</span> support</span>  </span></span>
 37</span>     return</span> retList</span>,</span> supportData</span></span>
 38</span>  </span></span>
 39</span> def</span> calSupport</span>(</span>D</span>,</span> Ck</span>,</span> min_support</span>)</span>:</span></span>
 40</span>     dict_sup</span> =</span> {</span>}</span></span>
 41</span>     for</span> i</span> in</span> D</span>:</span></span>
 42</span>         for</span> j</span> in</span> Ck</span>:</span></span>
 43</span>             if</span> j</span>.</span>issubset</span>(</span>i</span>)</span>:</span></span>
 44</span>                 if</span> not</span> j</span> in</span> dict_sup</span>:</span></span>
 45</span>                     dict_sup</span>[</span>j</span>]</span> =</span> 1</span></span>
 46</span>                 else</span>:</span></span>
 47</span>                     dict_sup</span>[</span>j</span>]</span> +=</span> 1</span></span>
 48</span>     sumCount</span> =</span> float</span>(</span>len</span>(</span>D</span>)</span>)</span></span>
 49</span>     supportData</span> =</span> {</span>}</span></span>
 50</span>     relist</span> =</span> [</span>]</span></span>
 51</span>     for</span> i</span> in</span> dict_sup</span>:</span></span>
 52</span>         temp_sup</span> =</span> dict_sup</span>[</span>i</span>]</span> /</span> sumCount</span></span>
 53</span>         if</span> temp_sup</span> >=</span> min_support</span>:</span></span>
 54</span>             relist</span>.</span>append</span>(</span>i</span>)</span></span>
 55</span> #</span> 此处可设置返回全部的支持度数据（或者频繁项集的支持度数据）</span></span>
 56</span>             supportData</span>[</span>i</span>]</span> =</span> temp_sup</span></span>
 57</span>     return</span> relist</span>,</span> supportData</span></span>
 58</span>  </span></span>
 59</span> #</span> 改进剪枝算法</span></span>
 60</span> def</span> aprioriGen</span>(</span>Lk</span>,</span> k</span>)</span>:</span></span>
 61</span>     retList</span> =</span> [</span>]</span></span>
 62</span>     lenLk</span> =</span> len</span>(</span>Lk</span>)</span></span>
 63</span>     for</span> i</span> in</span> range</span>(</span>lenLk</span>)</span>:</span></span>
 64</span>         for</span> j</span> in</span> range</span>(</span>i</span> +</span> 1</span>,</span> lenLk</span>)</span>:</span>  #</span> 两两组合遍历</span></span>
 65</span>             L1</span> =</span> list</span>(</span>Lk</span>[</span>i</span>]</span>)</span>[</span>:</span>k</span> -</span> 2</span>]</span></span>
 66</span>             L2</span> =</span> list</span>(</span>Lk</span>[</span>j</span>]</span>)</span>[</span>:</span>k</span> -</span> 2</span>]</span></span>
 67</span>             L1</span>.</span>sort</span>(</span>)</span></span>
 68</span>             L2</span>.</span>sort</span>(</span>)</span></span>
 69</span>             if</span> L1</span> ==</span> L2</span>:</span>  #</span> 前k-1项相等，则可相乘，这样可防止重复项出现</span></span>
 70</span>                 #</span> 进行剪枝（a1为k项集中的一个元素，b为它的所有k-1项子集）</span></span>
 71</span>                 a</span> =</span> Lk</span>[</span>i</span>]</span> |</span> Lk</span>[</span>j</span>]</span>  #</span> a为frozenset()集合</span></span>
 72</span>                 a1</span> =</span> list</span>(</span>a</span>)</span></span>
 73</span>                 b</span> =</span> [</span>]</span></span>
 74</span>                 #</span> 遍历取出每一个元素，转换为set，依次从a1中剔除该元素，并加入到b中</span></span>
 75</span>                 for</span> q</span> in</span> range</span>(</span>len</span>(</span>a1</span>)</span>)</span>:</span></span>
 76</span>                     t</span> =</span> [</span>a1</span>[</span>q</span>]</span>]</span></span>
 77</span>                     tt</span> =</span> frozenset</span>(</span>set</span>(</span>a1</span>)</span> -</span> set</span>(</span>t</span>)</span>)</span></span>
 78</span>                     b</span>.</span>append</span>(</span>tt</span>)</span></span>
 79</span>                 t</span> =</span> 0</span></span>
 80</span>                 for</span> w</span> in</span> b</span>:</span></span>
 81</span>                     #</span> 当b（即所有k-1项子集）都是Lk（频繁的）的子集，则保留，否则删除。</span></span>
 82</span>                     if</span> w</span> in</span> Lk</span>:</span></span>
 83</span>                         t</span> +=</span> 1</span></span>
 84</span>                 if</span> t</span> ==</span> len</span>(</span>b</span>)</span>:</span></span>
 85</span>                     retList</span>.</span>append</span>(</span>b</span>[</span>0</span>]</span> |</span> b</span>[</span>1</span>]</span>)</span></span>
 86</span>     return</span> retList</span></span>
 87</span> </span></span>
 88</span> def</span> apriori</span>(</span>dataSet</span>,</span> minSupport</span>=</span>0.2</span>)</span>:</span></span>
 89</span> #</span> 前3条语句是对计算查找单个元素中的频繁项集</span></span>
 90</span>     C1</span> =</span> createC1</span>(</span>dataSet</span>)</span></span>
 91</span>     D</span> =</span> list</span>(</span>map</span>(</span>set</span>,</span> dataSet</span>)</span>)</span>  #</span> 使用list()转换为列表</span></span>
 92</span>     L1</span>,</span> supportData</span> =</span> calSupport</span>(</span>D</span>,</span> C1</span>,</span> minSupport</span>)</span></span>
 93</span>     L</span> =</span> [</span>L1</span>]</span>  #</span> 加列表框，使得1项集为一个单独元素</span></span>
 94</span>     k</span> =</span> 2</span></span>
 95</span>     while</span> (</span>len</span>(</span>L</span>[</span>k</span> -</span> 2</span>]</span>)</span> ></span> 0</span>)</span>:</span>  #</span> 是否还有候选集</span></span>
 96</span>         Ck</span> =</span> aprioriGen</span>(</span>L</span>[</span>k</span> -</span> 2</span>]</span>,</span> k</span>)</span></span>
 97</span>         Lk</span>,</span> supK</span> =</span> scanD</span>(</span>D</span>,</span> Ck</span>,</span> minSupport</span>)</span>  #</span> scan DB to get Lk</span></span>
 98</span>         supportData</span>.</span>update</span>(</span>supK</span>)</span>  #</span> 把supk的键值对添加到supportData里</span></span>
 99</span>         L</span>.</span>append</span>(</span>Lk</span>)</span>  #</span> L最后一个值为空集</span></span>
100</span>         k</span> +=</span> 1</span></span>
101</span>     del</span> L</span>[</span>-</span>1</span>]</span>  #</span> 删除最后一个空集</span></span>
102</span>     return</span> L</span>,</span> supportData</span>  #</span> L为频繁项集，为一个列表，1，2，3项集分别为一个元素</span></span>
103</span> </span></span>
104</span> #</span> 生成集合的所有子集</span></span>
105</span> def</span> getSubset</span>(</span>fromList</span>,</span> toList</span>)</span>:</span></span>
106</span>     for</span> i</span> in</span> range</span>(</span>len</span>(</span>fromList</span>)</span>)</span>:</span></span>
107</span>         t</span> =</span> [</span>fromList</span>[</span>i</span>]</span>]</span></span>
108</span>         tt</span> =</span> frozenset</span>(</span>set</span>(</span>fromList</span>)</span> -</span> set</span>(</span>t</span>)</span>)</span></span>
109</span>         if</span> not</span> tt</span> in</span> toList</span>:</span></span>
110</span>             toList</span>.</span>append</span>(</span>tt</span>)</span></span>
111</span>             tt</span> =</span> list</span>(</span>tt</span>)</span></span>
112</span>             if</span> len</span>(</span>tt</span>)</span> ></span> 1</span>:</span></span>
113</span>                 getSubset</span>(</span>tt</span>,</span> toList</span>)</span></span>
114</span>  </span></span>
115</span> def</span> calcConf</span>(</span>freqSet</span>,</span> H</span>,</span> supportData</span>,</span> ruleList</span>,</span> minConf</span>=</span>0.7</span>)</span>:</span></span>
116</span>     for</span> conseq</span> in</span> H</span>:</span>  #</span>遍历H中的所有项集并计算它们的可信度值</span></span>
117</span>         conf</span> =</span> supportData</span>[</span>freqSet</span>]</span> /</span> supportData</span>[</span>freqSet</span> -</span> conseq</span>]</span>  #</span> 可信度计算，结合支持度数据</span></span>
118</span>         #</span> 提升度lift计算lift = p(a & b) / p(a)*p(b)</span></span>
119</span>         lift</span> =</span> supportData</span>[</span>freqSet</span>]</span> /</span> (</span>supportData</span>[</span>conseq</span>]</span> *</span> supportData</span>[</span>freqSet</span> -</span> conseq</span>]</span>)</span></span>
120</span>  </span></span>
121</span>         if</span> conf</span> >=</span> minConf</span> and</span> lift</span> ></span> 1</span>:</span></span>
122</span>             print</span>(</span>freqSet</span> -</span> conseq</span>,</span> '</span>--></span>'</span>,</span> conseq</span>,</span> '</span>支持度</span>'</span>,</span> round</span>(</span>supportData</span>[</span>freqSet</span>]</span>,</span> 6</span>)</span>,</span> '</span>置信度：</span>'</span>,</span> round</span>(</span>conf</span>,</span> 6</span>)</span>,</span></span>
123</span>                   '</span>lift值为：</span>'</span>,</span> round</span>(</span>lift</span>,</span> 6</span>)</span>)</span></span>
124</span>             ruleList</span>.</span>append</span>(</span>(</span>freqSet</span> -</span> conseq</span>,</span> conseq</span>,</span> conf</span>)</span>)</span></span>
125</span>  </span></span>
126</span> #</span> 生成规则</span></span>
127</span> def</span> gen_rule</span>(</span>L</span>,</span> supportData</span>,</span> minConf</span> =</span> 0.7</span>)</span>:</span></span>
128</span>     bigRuleList</span> =</span> [</span>]</span></span>
129</span>     for</span> i</span> in</span> range</span>(</span>1</span>,</span> len</span>(</span>L</span>)</span>)</span>:</span>  #</span> 从二项集开始计算</span></span>
130</span>         for</span> freqSet</span> in</span> L</span>[</span>i</span>]</span>:</span>  #</span> freqSet为所有的k项集</span></span>
131</span>             #</span> 求该三项集的所有非空子集，1项集，2项集，直到k-1项集，用H1表示，为list类型,里面为frozenset类型，</span></span>
132</span>             H1</span> =</span> list</span>(</span>freqSet</span>)</span></span>
133</span>             all_subset</span> =</span> [</span>]</span></span>
134</span>             getSubset</span>(</span>H1</span>,</span> all_subset</span>)</span>  #</span> 生成所有的子集</span></span>
135</span>             calcConf</span>(</span>freqSet</span>,</span> all_subset</span>,</span> supportData</span>,</span> bigRuleList</span>,</span> minConf</span>)</span></span>
136</span>     return</span> bigRuleList</span></span>
137</span>  </span></span>
138</span> if</span> __name__</span> ==</span> '</span>__main__</span>'</span>:</span></span>
139</span>     dataSet</span> =</span> data_translation</span></span>
140</span>     L</span>,</span> supportData</span> =</span> apriori</span>(</span>dataSet</span>,</span> minSupport</span> =</span> 0.02</span>)</span></span>
141</span>     rule</span> =</span> gen_rule</span>(</span>L</span>,</span> supportData</span>,</span> minConf</span> =</span> 0.35</span>)</span></span></code></pre>
例如{‘其他蔬菜’，‘酸奶’}=>{‘全脂牛奶’}支持度约为2.23%，置信度约为51.29%。说明同时购买酸奶、其他蔬菜和全脂牛奶这3种商品的概率达51.29%，而这种情况发生的可能性约为2.23%。综合输出结果分析，顾客购买酸奶和其他蔬菜的时候会同时购买全脂牛奶，其置信度最大达到51.29%。因此，顾客同时购买其他蔬菜、根茎类蔬菜和全脂牛奶的概率较高。</p>
2、模型分析</a></h3>
对于模型结果，从购物者角度进行分析:现代生活中，大多数购物者为“家庭煮妇”，购买的商品大部分是食品，随着生活质量的提高和健康意识的增加，其他蔬菜、根茎类蔬菜和全脂牛奶均为现代家庭每日饮食的所需品。因此，其他蔬菜、根茎类蔬菜和全脂牛奶同时购买的概率较高，符合人们的现代生活健康意识。</p>
3、模型应用</a></h3>
以上的模型结果表明:顾客购买其他商品的时候会同时购买全脂牛奶。因此，商场应该根据实际情况将全脂牛奶放在顾客购买商品的必经之路上，或是放在商场显眼的位置，以方便顾客拿取。顾客同时购买其他蔬菜、根茎类蔬菜、酸奶油、猪肉、黄油、本地蛋类和多种水果的概率较高，因此商场可以考虑捆绑销售，或者适当调整商场布置，将这些商品的距离尽量拉近，从而提升顾客的购物体验。</p>


航空公司客户价值分析
Unknown — Sun, 12 Mar 2023 21:54:00 +0800
import</span> pandas</span> as</span> pd</span></span>
import</span> numpy</span> as</span> np</span></span>
import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span>
import</span> seaborn</span> as</span> sn</span></span>
data</span>=</span>pd</span>.</span>read_csv</span>(</span>'</span>../air_data.csv</span>'</span>)</span></span>
data</span></span>
#</span>数据的描述性统计</span></span>
explore</span>=</span>data</span>.</span>describe</span>(</span>percentiles</span>=</span>[</span>]</span>,</span>include</span>=</span>'</span>all</span>'</span>)</span>.</span>T</span></span>
explore</span></span></code></pre>

Kirino 的技术档案

Grain用于读取和处理用于训练和评估 JAX 模型的数据

LeNet（Jax/Flax）实现

模型结构</a></h1> 总体来看，LeNet（LeNet-5）由两个部分组成：</p> 卷积编码器：由两个卷积层组成;</li>

LeNet（PyTorch）实现

模型结构</a></h1> 总体来看，LeNet（LeNet-5）由两个部分组成：</p> 卷积编码器：由两个卷积层组成;</li>

数据挖掘第6次作业

数据挖掘作业5

数据挖掘作业4

航空公司客户价值分析